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Preface 


Personnel  acquisition  planning  for  AWACS  crew  positions  is 
*  currently  based  on  "best  guesses"  driven  by  near  term  loss  expectations. 

For  this  reason,  managers  largely  restrict  themselves  to  dealing  with 
^  short  term  corrections  in  manning  level  deficiencies.  Long  term 

management  is  largely  a  matter  of  manager  intuition.  This  research  was 
conducted  to  extend  management  capabilities  to  handle  long  range 
acquisition  planning  in  a  structured  environment. 

The  process  of  this  research  was  fraught  with  difficulties  and 
dead  ends.  That  these  were  overcome  is  largely  due  to  the  guidance 
provided  by  my  advisor,  LtCol  Skip  Valusek  (u^o  I  think  could  teach 
Machiavelli  a  few  tricks). 

Finally,  I  owe  a  debt  of  thanks  to  my  wife  Robin,  for  her  support 
and  understanding  throughout  my  AFIT  experience,  and  a  debt  pure  and 
simple  to  my  children,  upon  whom  I  will  lavish  attention,  now  that  I  am 


done  here. 
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Abstract 


The  purpojSe  of  this  research  was  to  define  an  appropriate  tool  to 
assist  AWACS^'personnel  managers  in  determining  the  personnel 


acquisitions  that  are  required  for  AWACS  operations.  The  approach  used 
to  structure  the  decision  process  was  that  of  a  Decision  Support  System 
(DSS).  The  goal  of  the  specified  DSS  was  to  provide  an  initial  point  to 
better  state  personnel  acquisition  requirements,  particularly  with 
regard  to  the  experience  of  the  crew  force,  and  to  provide  a  structure 
for  improvement  in  the  DSS  itself. 

The  decisions  supported  by  the  designed  DSS  go  beyond  v/,hat  has 
been  attempted  oy  AiiACS  air  crew  managers  in  the  past,  explicitly 
addressing  experience  level  inventories  and  the  need  to  stabilize  both 
manning  levels  and  experience  levels  in  the  various  AWACS  crew 
positions.  The  overall  design  of  the  DSS  to  support  these  goals  is 
specified  through  the  mechanisms  of  concept  mapping  and  storyboarding, 
and  a  kernel  system  is  developed.  Adaptive  design  is  adopted  as  the 
means  for  continued  system  development  and  evolution.  '  ’  -  /  — 


A  CBCISION  9LPP0RT  SVSTCM  (DSB)  FOR 
A^MCS  PERSCMvEL  ACGUIBITION  l-WNAGEMENT 

I.  Introduction 

Background 

The  Airborne  Warning  and  Control  System  (AWACSl  is  an  air  battle 
management  system.  Its  functions  are  to  provide  airborne  surveillance 
and  to  control  friendly  aircraft  in  air  battles  in  its  surveillance 
area.  These  functions  are  tactical  in  nature;  therefore,  AWACS  belongs 
to  the  Tactical  Air  Command  (TAC),  with  the  28**'  Air  Division  (28  AD)  at 
Tinker  AFB  providing  primary  management  of  the  system. 

While  the  functions  of  AWACS  are  tactical,  the  personnel 
requirements  of  the  system  are  largely  alien  to  TAC.  TAC  is  primarily 
oriented  to  fighter  aircraft  with  one  or  two  rated  crew  positions. 

AWACS,  in  contrast,  has  12  crew  positions  (Table  1),  only  two  of  which 
are  rated.  Managing  this  diverse  crew  force  requires  the  ability  to 
plan  new  personnel  acquisitions  for  each  crew  position  with  enough 
accuracy  to  keep  the  crew  force  "stable"  (i.e.,  maintain  manning  and 
experience  levels  near  nominal  values).  One  of  the  consequences  of  the 
TAC/AWACS  personnel  orientation  mismatch  is  an  absence  of  management 
tools  to  support  the  personnel  acquisition  planning  process  across  the 
entire  spectrum  of  AWAC^  crew  positions.  It  is  this  absence  of 
management  tools  in  the  AWACTS  personnel  management  system  which  is  the 


concern  of  this  study. 


Table  1  —  AUIACS  Crew  Cfjxnpcxient  Information 


Author izat ions 


FI laht  Crew  Positions 

#/Crew 

Line 

Staff 

1.  Pilot*  (P/CP) 

2 

103 

18 

2.  Navigator*  (f^V) 

1 

50 

14 

3.  Flight  Engineer  (FE) 

1 

55 

3 

Mission  Crew  Positions 

4.  Mission  Crew  Commander  (MCC) 

1 

78 

28 

5.  Senior  Director  (3D) 

1 

50 

9 

6.  Weapons  Director  (WD) 

4 

238 

8 

7.  Air  Surveillance  Officer  (ASD) 

1 

54 

8 

8.  Air  Surveillance  Technician  (AST) 

3 

219 

23 

9.  Communications  System  Operator 

(CSO) 

1 

101 

6 

10.  Communications  Technician 

(CT) 

1 

50 

6 

11.  Computer  Maintenance  Techr  ician 

(CDMT) 

1 

50 

6 

12.  Airborne  Radar  Technician 

(ART) 

2 

100 

5 

• 

Rated  Positions 

•  • 

Mod  30- 

-35 

Fyoqrammed  Flying  Training 

The  Programmed  Flying  Training  (PFT)  program  is  a  personnel 
management  effort  originally  designeJ  *‘o  allow  major  commands  to 
identify  new  requirements  for  rated  officers  to  the  Air  Training  Command 
(ATC).  It  was  later  expanded  to  cover  non-rated  crew  positions.  The 
stated  goal  of  the  PFT  program  is  to  "sustain  an  iriventory  ...  to  mec't 
reouirements  ^ile  retaining  a  credible  combat  posture"  CDept  of  the  Air 
Force,  AF  Planning  Document,  Change  6  :  31.  For  any  weapons  system 
mannE?d  through  the  PFT,  this  amounts  to  maintaining  each  crew  position 
at  an  adequate  manning  level  with  an  experience  profile  that  is 
SLiffic lent  to  ensure  the  system  is  effective.  Assuming  ATC  can  provide 
the  training  capacity  to  fill  the  requirements  identified  by  the  major 
commands,  the  PFT  may  be  capable  of  achieving  its  stated  goals. 

However,  regardless  of  ATC  capabilities,  the  PFT  can  only  function 
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properly  if  the  requiremeritB  identified  by  the  mAjor  c-Dmmands  are 
accurate.  Accurate  requirements  identification  requires  that  personnel 
managers  have  adequate  management  tools  that  allow  them  to  properly 
identify  personnel  needs.  Thiese  tools  have  never  been  developed  fully, 
part icL'larly  for  the  non-rated  positions.  Major  factors  in  the  lack  of 
development  of  management  tools  for  the  non-rated  posit  lo  have  been 
the  lack  of  good  criteria  for  measuring  the  experience  level  of  the 
force  and  the  specialized  (small  sized)  nature  of  the  positions. 

flWACS  and  the  PFT.  Li.der  the  current  AWACS  PTT  process,  the  28  AD 
hosts  annual  PTT  conferences  and  quarterly  review  conferences  at  Tinker 
AFB  to  plan  new  crew  acquisitions  for  a  five  year  period.  Participants 
are  from  TAC,  the  Air  Force  Military  Personnel  Center  (AFM=C)  and  the  28 
AD.  These  three  participants  produce  a  consensus  figure  for  the 
appropriate  n«jmber5  of  new  acquisitions  required  to  keep  the  AWACS  crew 
force  healthy  with  the  constraint  that  the  desired  acquisition  is 
achievable.  (By  achievable,  it  is  meant  that  constraints  that  bind  each 
organization  in  the  acquisition  process  are  not  violated,  e.g.,  more 
bodies  are  not  "bought"  than  TAC  or  AFMPC  can  afford,  or  more  are  not 
bought  than  the  28  AD  can  give  final  training  to  in  a  given  acquisition 
eye  le) . 

The  primary  28  AD  organization  that  deals  with  PFT  issues  consists 
of  representatives  from  1)  the  552  Tactical  Training  Squadron  (TTS)  and 
the  966  Airborne  Warning  and  Control  Training  Squadron  (AWACTS),  to 
cover  28  AD  controlled  training  issues  (i.e.,  annual  capacities, 
changes,  etc.),  and  2)  the  personnel  nvanagers  for  each  crew  position, 
v,>ho  are  tasked  with  providing  the  basic  information  on  uhich  i,he  uhole 
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PFT  conference  outcome  is  based,  i.e.,  identifying  the  necessary  new 
acquisitions  needed  to  keep  their  respective  crew  positions  "healthy". 

For  AWACS,  the  before-mentioned  lack  of  appropriate  personnel 
management  tools  has  resulted  in  managers  determining  personnel 
requirements  through  the  use  of  individually  developed,  unstructured 
heuristics.  Each  of  the  crew  position  managers  has  his  own  "rules"  for 
arriving  at  a  best  guess;  these  rules  are  not  stated  anywhere,  and 
change  every  time  the  manager  for  the  position  changes.  For  these 
managers,  estimating  needs  is  typically  an  exercise  in  maintaining  a 
personal  knowledge  of  all  members  of  the  crew  force  with  respect  to  the 
factors  that  impact  the  manning  status  of  the  crew  position  (factors 
such  as  PCSs,  separat ions,  upgrades,  ratings  on  evaluations,  etc).  This 
information  is  "filed"  by  each  manager  in  his  own  unique  format  for  his 
sole  use.  Fundamentally,  the  personnel  requirements  determination 
process  is  based  on  the  manager’s  intuition. 

Given  the  diversity  of  factors  that  impact  the  status  of  a  crew 
position  (discussed  in  the  next  section)  and  manager's  lack  of  tools,  it 
15  not  surprising  that  the  personnel  requirements  identified  by  the 
managers  are  driven  by  short  range  considerations.  Consequently,  the 
crew  positions  suffer  from  severe  long  term  fluctuations  in  total 
manning  and  in  experience  base.  The  manning  fluctuations  shown  in 
Figure  1  are  representative  of  fluctuations  that  have  been  detected  in 
the  AWACS  crew  force  in  past  analyses  of  the  crew  force  CStonel.  The 
shortfall  from  the  authorized  manning  level  is  plotted  at  six  month 
intervals.  Of  special  note  are  the  severe  peaks  at  five  year  intervals. 
Under  current  planning  processes,  these  peaks  are  built  into  the  system. 
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PERCENT  SHORTFALL  FROM  AUTHORIZED  LEVEL 


82-1  83-1  84-1  85-1  8B-1  87-1 


82-2  83-2  84-2  85-2  88-2  87-2 

SEMI  ANNUAL  INCREMENTS 

Figure  1  —  Crew  Position  Manning  Fluctuations  (representat ive) 

and  are  repeated  every  five  years. 

A  comparable  graph  for  experience  level  is  not  directly  available, 
due  to  the  lack  of  experience  level  criteria  for  most  of  the  crew 
positions.  However,  it  is  instructive  to  look  at  the  proportion  of  a 
crew  position  that  is  made  up  of  first  term  personnel  (for  positions 
that  acquire  personnel  through  new  Air  Force  accessions),  as  an 
indicator  of  the  level  of  experience.  For  crew  positions  such  as  the  WD 
position,  there  is  a  direct  correlation  between  the  periods  of  high 
manning  in  Figure  1  and  a  high  representation  of  first  term  personnel  in 
the  crew  position.  This  is  to  be  expected,  since  acquisitions  for  this 
position  have  historically  been  made  up  of  personnel  new  to  the  Air 
Force.  A  representative  graph  for  this  condition  is  shown  in  Figure  2. 
As  in  the  previous  figure,  significant  fluctuations  in  the  data  of 
interest  (assuming  the  proportion  of  first  term  personnel  is  accepted  as 
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PERCENT  OF  FIRST  TERM  PERSONNEL  IN 

CREW  POSIT  ION 


I  SEMI  ANNIML  INCREMENTS 

Figure  2  —  First  Term  Personnel  in  AWACS  Crew  Positions  (representative) 


an  indicator  of  overall  experience  in  the  crew  position)  are  observed. 

AHACS  F^sonnel  Structure 

AWACS’  mission  obligations  and  capabilities  are  determined  by 
higher  headquarters,  and  result  from  specific  defense  objectives 
(ranging  from  NATO  support  to  showing  the  flag  in  hot  spots  around  the 
world).  Specific  personnel  authorizations  for  the  system  depend  on  the 
extent  of  the  mission  for  i«*nich  the  system  is  responsible.  The  number 
of  authorizations  an  air  crew  manager  deals  with  varies  by  crew 
position.  This  number  has  four  primary  factors: 

1)  the  number  of  members  required  in  each  position  for  a  full 

crew; 

2)  the  number  of  overhead  positions  authorized  for  staff 
functions; 


3)  the  number  of  crews  authorized  per  primary  assigned  aircraft 
(P^¥i),  and; 


6 


4)  the  number  of  PAA. 

Keeping  these  author izat ions  filled  is  a  basic  part  of  the  air  crew 
‘  manager’s  function.  The  personnel  acquisition  requirements  he 

determines  for  the  PFT  are  driven  at  a  very  basic  level  by  the  need  to 
,  predict  u*ien  some  of  these  authorizations  will  become  vacant.  However, 

he  must  also  deal  with  changes  in  authorizations.  All  four  of  the 
factors  that  drive  authorizations  are  subject  to  change.  The  creation 
of  new  units  requires  new  staff  authorizations  (e.g.,  the  creation  of 
the  962  AWACS  at  Elmendorf  AFB  created  new  staff  authorizations  to 
provide  the  squadron  staff)  and  changing  missions  can  alter  crew 
compositions,  PAA  and  crew/PAA  ratio  (e.g.,  the  transition  of  airframes 
from  Block  20-25  to  Block  30-35  configuration,  reflecting  mission 
requirements  for  more  surveillance  and  control  capacity.  The  change  in 
mission  will  require  authorization  increases  of  more  than  25"/.  for  both 
the  WD  and  AST  positions). 

The  AWACS  crew  positions  were  presented  in  Table  1.  As  stated, 
there  are  12  crew  positions;  all  are  required  for  AWACS  to  perform  its 
mission.  The  current  authorizations  for  each  crew  position  are  also 
presc?nted  in  Table  1.  Factors  to  note  in  the  table  are  1)  the  absolute 
differences  in  number  of  personnel  being  managed  in  each  crew  position, 
2)  the  number  of  personnel  required  per  crew  in  the  position,  and  3)  the 
size  of  the  staff  authorization  for  each  position.  These  factors  are  of 
interest  because  they  indicate  the  differences  in  problem  scope  faced  by 
the  individual  crew  position  managers  who  collectively  are  required  to 
keep  the  AWACS  crew  force  healthy  and  the  system  mission  effective.  For 
(1),  the  number  of  authorizations  in  different  crew  positions  vary  by  a 
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factor  of  close  to  five;  some  managers  have  many  more  people  with  whom 
they  must  maintain  contact  under  current  management  methods.  For  (2), 
the  number  of  authorizations  per  crew  position  per  crew  indicates  that 
some  positions  have  more  leeway  in  their  manning  tolerances  than  others. 
This  is  also  the  case  for  (3),  since  positions  with  large  staff 
authorizations  have  reserve  personnel  in  the  system  to  perform  flying 
missions  if  shortages  occur).  These  issues  are  discussed  more 
extensively  in  the  following  section. 

Diversity  of  Factors  among  Crew  Positions.  All  of  the  AWACS  crew 
positions  must  remain  healthy  for  the  system  to  perform  its  mission 
effectively.  However,  the  considerations  facing  each  position  manager 
to  achieve  a  healthy  status  for  his  position  are  not  the  same.  A 
contrast  of  Pilots  and  Weapons  Directors  provides  valuable  insights  into 
the  differences  in  the  factors  that  affect  each  crew  position. 

Pilots  have  well-defined  performance  factors  that  indicate  the 
competence  of  individual  crew  members  and  that  can  be  aggregated  to 
indicate  the  experience  level  of  the  AWACS  pilot  crew  force.  Measures 
such  as  total  flying  hours,  number  of  landings,  number  of  touch-and-goes 
and  number  of  air-refuelings  accomplished  provide  a  direct  measure  of 
the  pilot's  experience  because  the  pilot  is  exercising  the  skills  that 
affect  his  experience  level  for  every  measure  mentioned.  The  pilot 
manager  has  reasonable  measures  for  determining  the  experience  profile 
of  the  crew  position  he  manages. 

In  the  numbers  game,  however,  the  pilot  manager  has  severe 
problems  predicting  losses  over  time.  Pilots  have  fairly  extensive  non- 
AWACS  assignment  opportunities  <e.g.,  staff  and  command  Jobs  and  flying 
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jobs  on  other  airframes  are  filled  by  AF>PC  with  AWACS  pilots).  With 
pilots  in  short  supply  Air  Force  wide,  AFT1=C  juggles  its  supply  of 
pilots,  often  giving  the  losing  organizations  short  notice.  More 
critical  to  the  pilot  manager  however,  is  the  demand  for  the  pilot's 
skills  in  the  civilian  sector.  The  myriad  of  factors  that  change  the 
demand  for  pilots  in  the  civilian  sector  changes  the  "leak  rate"  the 
manager  must  contend  with,  and  generally,  he  has  no  way  of  predicting 
these  changes  effectively. 

The  story  for  the  WD  manager  is  much  different.  In  the  experience 
profile  area,  as  an  example,  even  though  total  flying  hours  are  logged 
for  WDs  as  they  are  for  pilots,  flying  hours  may  be  a  poor  measure  of 
the  individual's  competence  for  the  following  reasons: 

1)  the  time  spent  by  the  WD  exercising  his  primary  airborne 
skills,  i.e. ,  controlling  other  aircraft,  may  only  be  a  small  fraction 
of  the  total  time  he  spends  in  the  air;  it  is  not  uncommon  for  the  WD  to 
control  no  aircraft  due  to  factors  such  as  weather  and  AWACS  systems 

mal funct ions; 

2)  the  'work  load  an  individual  WD  experiences  is  not  uniform 
because  of  different  mission  profiles;  and 

3)  since  there  are  multiple  WDs  on  a  given  crew,  the 
accumulation  of  experience  will,  in  general,  not  be  uniform;  for  a 
difficult  control  problem,  the  MCC  may  have  his  most  experienced 
operator  be  in  charge  of  directing  the  problem. 

This  lack  of  good  experience  measures  has  resulted  in  the  current 
practice  in  the  AWACS  community  of  designating  WDs  as  fully  experienced 
after  they  have  been  flying  for  one  year  following  graduation  from 
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flight  training  at  the  552  TTS.  (In  contrast,  non-airborne  WDs, 
operating  at  ground  TACCs,  are  rated  for  experience  by  the  number  of 
controls  they  direct.  With  current  data  availability  constraints,  this 
is  not  an  option  for  rating  the  AWACS  WD  force). 

In  the  manning  level  arena,  however,  the  WD  manager  is  faced  with 
a  crew  force  that  has  limited  non-AWACS  assignment  opportunities  and 
almost  no  demand  in  the  civilian  sector  for  its  military  skills.  Losses 
are  much  more  predictable  over  a  longer  period  of  time  in  the  WD  crew 
force  than  they  are  in  the  pilot  crew  force. 

In  general,  the  management  of  each  crew  position  entails 
consideration  of  a  unique  mix  of  factors.  Some  of  the  critical  factors 
that  have  differing  impacts  on  the  different  crew  positions  are: 

1)  Non-AWACS  assignment  opportunities  —  the  personnel  system  is 
closed  for  some  crew  positions  (ARTs,  AST)  and  crew  losses  are  mostly 
due  to  separations  (v^ich  are  fairly  predictable),  while  the  system  is 
open  for  others  <P,  FE,  CDMT)  and  short  notice  PCS  losses  are  a  major 
factor  in  disrupting  the  position's  manning  stability.  (Closed  and  open 
refer  to  v^ether  the  AFSC  for  a  particular  crew  position  is  "unique"  to 
AWACS.  If  the  AFSC  is  closed,  then  the  personnel  in  it  are  "trapped"  in 
the  AWACS  personnel  system) . 

2)  Experience  measures  —  some  crew  positions  have  well-defined 
measures  for  determining  experience  of  its  members  (e.g.,  Flight  Deck), 
others  have  less  well-defined  measures  (CD^TT,  ART),  but  operate  in  a 
technical  area  that  lends  itself  to  rather  unambiguous  measures  (repair 
record),  and  still  others  have  arbitrarily  defined  measures  that  provide 
no  guidance  (WD). 
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3)  Redundancy  in  the  crew  position  —  AWACS  mission  capabilities 
are  sensitive  to  crew  positions.  Crew  positions  that  are  manned  one 
deep  per  crew  can  degrade  system  performance  more  rapidly  than  crew 
positions  having  multiple  members  per  crew.  If  there  are  insufficient 
FEs,  the  E-3  will  never  get  off  the  ground;  if  there  are  insufficient 
WDs,  the  system  can  still  operate  at  some  degraded  level  with  a  crew 
that  is  undermanned  in  the  WD  position. 

4)  Crew  position  independence  —  manning  of  many  of  the  crew 
positions  can  be  managed  totally  independently  of  all  of  the  other 
positions;  the  manager's  only  consideration  is  the  health  of  the 
position  he  manages-  Other  positions  are  not  independent.  For  example, 
a  significant  portion  of  the  MCC  force  is  drawn  from  the  SD  population, 
v^ile  almost  all  SDs  come  from  the  WD  position.  Therefore,  the  WD 
manager  may  be  faced  with  a  deficiency  in  WD  manning  because  of  a 
decision  outside  of  his  control  to  rob  the  WD  personnel  pool  to  fix  the 
SD  pool.  Further,  this  deficiency  is  generally  not  only  a  numbe?rs  issue 
but  involves  the  WD  experience  base  because  the  most  experienced  UDs  are 
selected  for  upgrade  to  the  SD  position. 

Fundamentally,  the  structure  of  the  crew  is  non-homogeneous;  crew 
positions  can  be  grouped  into  different  categories  (flight  —  mission; 
rated  —  non-rated;  officer  —  enlisted),  and  each  category  imposes 
special  considerations  on  how  personnel  are  managed.  Regardless  of 
differences  in  the  positions,  all  should  be  managed  with  equal  skill. 

Problem  Statement 

If  AWACS  is  to  be  as  effective  as  possible  in  its  mission,  it  must 
be  properly  manned.  The  goal  of  the  PFT  is  to  achieve  this  manning. 
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quantitatively  and  qualitatively,  in  such  a  manner  that  the  crew  force 
is  stable.  With  the  current  lack  of  management  tools,  AWACS  managers 
have  not  been  able  to  realistically  determine  their  manning  needs. 
Consequently,  the  AWACS  crew  force  has  been  chronically  unstable,  both 
in  absolute  numbers  and  in  the  experience  base  of  the  personnel 
constituting  the  crew  force.  If  AWACS  is  to  be  properly  manned,  the  air 
crew  managers  require  tools  to  integrate,  present,  manipulate  and 
analyze  information  that  is  pertinent  to  the  personnel  acquisition 
process. 

Objectives 

To  approach  the  design  of  a  tool  to  assist  in  accurately  predict 
manning  acquisition  requirements  for  the  PTT  effort.  To  meet  this 
objective,  the  following  sub-objectives  must  be  attained: 

1)  Establish  a  "management  paradigm"  for  determining  needs  that 
is  simple,  effective  and  uniform  for  all  air  crew  managers,  regardless 
of  the  idiosyncrasies  of  the  position  structure; 

2)  Identify  factors  impacting  manning  level; 

3)  Identify  factors  impacting  experience  level; 

4)  Identify  utiere  appropriate  data  is  located; 

5)  Determine  appropriate  models  for  analyzing  the  data,  and; 

6)  Provide  an  adaptive  design  "road  map"  so  the  tool  remains 
viable  in  changing  circumstances. 

The  following  chapters  will  discuss  the  methods  for  accomplishing 
the  above  steps,  the  specific  design,  conclusions  reached  during  the 
research  effort,  and  recommendations  for  further  areas  of  research  on 
this  subject. 
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II.  METHJXUDGY 


Suitability  of  a  Decision  Support  System  CD6S)  Approach 

The  problem  the  AWACS  managers  face  in  determining  personnel 
requirements  is  not  "well  structured".  Each  manager  deals  with  factors 
impacting  manning  that  are  common  to  all  positions,  and  with  factors 
unique  to  his  own.  Some  factors  are  dynamic,  and  possibly  worse,  the 
importance  of  factors  are  also  dynamic.  Because  of  the  diversity  of 
factors  and  their  dynamic  nature,  the  users  of  the  system  to  be  designed 
(crew  position  personnel  managers)  cannot  provide  a  functional 
specification  of  the  decision  process  to  be  supported.  Keen  argues  that 
the  user's  inability  to  provide  a  functional  specification  is  one  of  the 
hallmarks  of  a  process  for  ubich  a  DSS  is  appropriate  [Keen  ;  151.  The 
lack  of  a  functional  specification  prohibits  the  use  of  a  purely 
prescriptive  modeling  approach;  the  problem's  nature  is  such  that  an 
interactive,  semi -structured  process  is  called  for. 

Current  practices  in  the  AWACS  personnel  requirements  process  have 
never  emphasized  long  term  effects  of  the  factors  mariagers  consider  in 
making  their  manning  decisions.  The  impact  of  different  factors  on 
short  term  manning  profiles  are  reasonably  well  understood  by  the 
managers  —  in  most  instances,  the  short  term  condition  of  the  force  is 
all  that  is  managed.  However,  to  be  effective  in  stabilizing  the  crew 
force,  managers  must  understand  the  long  term  impact  of  their 
acquisition  decisions.  They  must  understand  the  long  term  impact  of 
managing  crew  positions  for  short  term  health. 

Any  system  that  implements  a  capability  to  examine  the  effects  of 
current  management  decisions  on  the  future  condition  of  the  crew  force 
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can  help  the  manager  understand  Icjng  term  effects  of  his  decisions.  In 
particular,  a  DSS  is  an  ideal  tool  in  this  respect,  since  it  can 
implement  the  desired  capability  in  a  system  that  is  designed  to 
"facilitate  the  ...  Cdecisionl  process  rather  than  to  force-fit  it  into 
a  designer's  notion  of  the  best  process"  [Young  :  1].  This  allows  the 
user  to  focus  on  the  problem,  further  clarifying  his  understanding  of  it 
(possibly  gaining  new  insights),  and  make  decisions.  This  is  a  distinct 
departure  from  traditional  "support  systems"  where  the  user  is  often 
consumed  by  the  mechanics  of  the  system,  to  the  detriment  of  process 
understanding.  Traditional  systems  design  is  a  "builder"  driven 
process;  as  a  result,  systems  are  designed  to  support  the  builder,  not 
the  user. 

A  further  reason  for  choosing  a  DSS  approach  is  the  necessarily 
evolutionary  development  of  the  management  system.  Changes  in  any 
system  developed  will  have  to  be  made,  since: 

1)  the  importance  of  factors  will  change  as  corrections  are  made 
in  the  force  structure,  with  new  factors  requiring  consideration  and 
some  current  factors  becoming  unimportant; 

2)  the  current  understanding  of  factor  importance  is  such  that 
mistakes  will  be  made.  The  system  will  need  to  be  modified  to  adjust 
for  the  user's  better  understanding  of  his  needs;  and 

3)  the  use  of  the  system  will  change  the  user's  decision  process 
from  uhiatever  process  is  first  "captured"  in  the  DSS.  The  DSS  will  need 
modification  to  support  new  processes  its  use  engenders. 

A  DSS,  oriented  around  adaptive  design,  provides  a  structured  framework 
to  evolve  the  system  being  developed. 


Adaptive  design  is  a  systems  design  approach  where  the  philosophy 
is  "start  small  and  grow"  CValusek  :  105  -  111].  Adaptive  design  shares 
the  elements  of  the  traditional  systems  design  approach,  i.e. , 
requirements  analysis,  design,  development  and  implementation.  However, 
in  contrast  to  the  traditional  approach,  it  does  not  fix  a  rigid  systems 
specification  before  development  starts.  Recognizing  the  user  usually 
cannot  fully  specify  the  system  desired  before  development  starts,  the 
adaptive  approach  stresses  putting  a  kernel  system  into  the  user's  hands 
and  evolving  it  as  a  result  of  the  user’s  reactions. 

Adaptive  design  is  critical  if  the  decision  process  being 
considered  is  to  be  effectively  supported.  Keen  provides  a  succinct 
graphic  that  describes  why  this  is  so  (Figure  3)  [Keen  :  16].  The  three 
elements  in  a  system  design  process  are  the  user,  builder  and  the  system 


Figure  3  —  An  Adaptive  Framework  for  DSS 


Atf«pt«d  fro«  CK««n  i  163 

being  developed.  In  design  processes  to  which  the  DSS  approach  is 
suited,  there  are  necessary  two  way  linkages  between  all  th^ee  elenents. 
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"A  system  is  a  "DSS"  only  if  each  _  is  relevant  to  the  situation" 

CKeen  ;  171. 

For  the  decision  proce  3  discussed  here,  all  linkages  are 
necessary  for  the  production  of  a  functional  system.  The  System  -  User 
linkages  are  required  because  the  AUJACS  manning  determination  process  is 
not  well  structured.  Users  will  gain  insights  as  the  system  starts  to 
structure  the  process;  the  user  will  therefore  change  his  use  of  the 
system.  The  User  -  Builder  linkages  are  required  to  ensure  the  proper 
system  is  implemented.  The  middle-out  design  approach  enhances 
communication  by  placing  a  kernel  system  in  the  user's  hands,  providing 
a  design  element  that  can  be  used  by  both  the  user  and  builder  to  define 
the  problem.  Proper  definition  of  the  problem  facilitates  system 
implementation.  Finally,  the  System-Builder  linkages  must  exist.  The 
altered  use  of  the  system  seen  in  the  User-System  loop  can  only  progress 
so  far  before  the  system  requires  adaptation  to  support  the  new  process. 
The  builder  then  provides  new  functions  to  the  system,  and  the  cycle 
starts  over  again. 

Expanding  on  the  above  theme,  Ackoff  makes  the  following 

observation  about  designing  support  systems; 

It  must  be  assumed  that  the  system  that  is  being  designed  will  be 
deficient  in  many  significant  ways.  Therefore  it  is  necessary  to 
identify  the  ways  in  which  it  may  be  deficient,  to  design 
procedures  for  detecting  its  deficiencies,  and  for  correcting  the 
_>ystem  so  as  to  remove  or  reduce  them.  Hence  the  system  should  be 
designed  to  be  flexible  and  adaptive.  ...  No  completely 
computerized  system  can  be  as  flexible  and  adaptive  as  can  a  man- 
machine  system.  CAckoff  :  1551 

The  key  points  here  are  1)  that  the  system  must  allow  for  evolution  if 
it  IS  to  be  effective,  and  2)  that  systems  that  are  designed  as  man- 
machine  systems  are  more  capable  of  adaptation  than  are  purely 
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mechanical  ones.  Adaptive  design  in  the  DSS  context  .s  highly  attuned 
to  both  of  these  factors.  It  requires  an  evolutionary  framework  for  the 
sys-en.  that  is  part  of  the  system  and  can  easily  allow  for  the  framework 
in  the  man-machine  interface  of  the  support  system. 

An  adaptive  design  appy-oach  also  has  "immediate  gratification" 
attributes  that  make  it  particularly  desirab’ e  Unen  contrasted  to 
traditional  systems  development  approaches.  Gome  of  these  attributes 
are; 

1)  Short  response  times  to  user  inputs.  Where  problem 
understanding  is  "foggy",  extracting  the  process  from  the  users  is  a 
time  intensive  process.  If  the  user  is  to  make  a  time  commitment,  he 
must  see  a  return  in  the  short  term. 

2)  User  part icipat ion  in  the  design  of  his  system.  The  user 
probably  is  aware  of  how  foggy  the  problem  is.  If  so,  he  kncws  mistakes 
will  be  made.  FV-oviding  a  system  that  is  responsive  to  his  changing 
perceptions,  many  of  uhich  will  result  from  the  process  of  defining  the 
problem,  allows  the  user  to  "get  the  system  he  wants"  not  the  one  he 
thought  he  wanted  (or  the  designer  thought  he  wanted).  The  pror ass  of 
watching  the  system  evolve,  due  to  his  inputs,  assures  the  user  that  the 
system  will  he  i.^at  he  cesires. 

3)  User  perception  that  he  is  being  understood.  As  a  corollary 

of  (2),  the  communications  between  the  "owner"  of  a  decision  process  and 
the  builder  of  the  support  tool  will  be  poor  because  of  the  different 
realms  of  expertise.  Mistakes  will  occur  because  of  poor  communicaticn 
even  in  problems  that  are  well  understood  by  the  user.  According  to 
Cerveny,  et  al ,  "the  use  of  the  traditional  [systems  designl  approach 
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does  not  adequately  address  the  underlying  issue  of  poor  communicaf ' cn 
between  the  user  and  the  analyst"  CCerveny,  et  al  :  543.  Adaptive 
design  pro\'ides  a  framework  for  identifying  mis-communi  cat  ions  and 
correcting  them.  The  feedback  in  the  communications  process,  indicated 
by  evolutionary  changes  in  the  system  being  designed,  reassures  the  user 
that  he  is  getting  >^at  he  wants. 

The  V/Ehicles  of  Adaptive  Design.  Adaptive  design,  as  being 
researched  at  the  Air  Force  Institute  of  Technology  (AFIT),  above  all, 
provides  a  means  for  structuring  the  development  of  support  systems. 

For  the  purposes  of  this  study,  three  major  structuring  devices  were 
ejrployed.  These  devices  a^e  concept  mapping,  storyboarding  via  RQMC, 
and  the  "hook  book". 

Concept  Mapping.  The  first  stage  in  the  adaptive  design 
process  is  developing  some  understanding  of  the  decision  process  being 
supported.  Concept  mapping  is  an  educational  tool  that  is  adaptable  to 
developing  this  understanding  CValusek  :  1073  It  provides  a  medium  for 
identifying  elements  of  the  decision  process  and  describing  the 
relations  between  the  elements.  With  a  map  of  the  decision  elements, 
the  builder  and  user  can  pick  an  appropriate  "initial  kernel"  (subset) 
system  to  be  developed.  (However,  the  actual  kernel  that  provides  the 
seed  for  the  system  to  be  developed  is  usually  not  fully  specified  until 
after  storyboarding  [described  in  the  next  paragraph!  is  complete). 

Storyboarding.  After  concept  mapping,  the  initial  kernel 
system  is  described  pictorial ly  on  paper  to  ensure  that  the  kernel  is 
what  the  user  actually  desires.  This  pictorial  description  of  the 
system  is  called  "storyboarding"  and  was  preposed  by  Andriole  as  a 
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technique  for  capturing  the  users  views  on  the  important  aspects  of  the 
decision  process  being  supported  CAndriole  :  463-469].  Storyboards,  in 
the  adaptive  design  realm,  provide  the  specification  of  the  system  to  be 
designed. 

The  process  of  storyboarding,  with  its  graphical  presentation  of 
the  DSS,  helps  clarify  the  decision  process  layout,  and  usually  leads  to 
iterative  changes  in  the  initial  kernel  that  is  identified  for 
development.  At  the  point  where  the  user  is  satisfied  that  the 
storyboards  reflect  his  desired  system,  the  "final  kernel"  has  been 
identified.  System  development  then  starts,  and  further  adaptation  of 
the  kernel  can  proceed  until  the  desired  system  is  evolved. 

RDMC.  In  conjunction  with  storyboarding,  the 
Representation,  Operations,  Memory  aids  and  Control  (ROMC)  structures 
described  by  Sprague  and  Carlson  provide  a  valuable  method  of  ensuring  a 
storyboard  design  is  complete  [Sprague,  et  al  :  103-118]. 

Representations  are  the  presentation  of  the  information  the  manager 
requires.  The  focus  is  on  what  form  the  presentation  takes  to  convey 
the  required  information.  Operations  are  the  functions  executed  by  the 
DSS  to  provide  representations.  The  focus  is  on  how  the  decision 
maker's  information  is  produced.  Memory  aids  are  the  mechanisms 
required  to  ensure  the  user  knows  v^ere  he  is  and  what  he  has  considered 
in  the  decision  process.  The  focus  is  on  providing  orientations  for  the 
decision  process.  Finally,  Control  mechanisms  are  the  mechanisms  that 
allow  the  user  to  direct  the  functioning  of  the  DSS.  The  focus  is  on 
facilitating  system  use.  All  four  o*  the  ROMC  factors  must  be 
consciously  accounted  for  in  the  design  i f  an  implemented  system  is  to 
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be  functional. 


The  "hook  book".  Finally,  the  system  must  have  a  mechanism 
facilitate  adaptation.  A  method  for  identifying  new  requirements  and 
system  deficiencies,  that  is  readily  available  whiie  the  Bystem  is  in 
use,  is  necessary  if  the  system  is  to  adapt  rapidly.  A  "hook  book", 
described  by  Valusek,  provides  the  necessary  mechanism  for  determining 
new  systems  requirements  C Valusek  :  1093. 

The  hook  book  is  a  software  facility  embedded  in  the  implemented 
system  that  is  accessible  from  any  point  in  the  system.  Fundamentally 
it  is  an  on-line  memory  aiding  facility  for  capturing  user  thoughts 
about  the  system  (or  the  decision  being  supported).  The  notes  to  be 
captured  are  structured  (see  Figure  4),  and  have  four  distinct  parts, 
these  being; 

1 )  the  date  of  the  note  entry; 

2)  a  label  for  the  entry; 

3)  a  brief  description  of  the  idea  that  prompted  the  note,  and; 

4)  the  circumstances  that  led  to  the  idea. 


DATE:  _  LABEL: 

IDEA;  _ 


CIROJMBT^VvCES: 


Figure  4  —  Hook  Book  Entry  Format 

Adapted  free  CVaiueeh  i  lOf] 

Each  part  serves  a  specific  purpose.  The  date  allows 
chronological  sorting  of  ideas,  the  label  allows  sorting  of  ideas  by 
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system  function,  the  description  provides  the  basic  memory  aid  as  to 
utiat  the  user  wants,  and  the  circumstances  provide  the  key  to  "detailed 
recall  of  the  Cuser’s3  idea  during  requirements  elicitation"  CValusek  : 
1093.  Together,  the  four  parts  of  the  hook  book  are  sufficient  for  the 
orderly  identification  of  new  system  requirements,  and  ensures  the 
system  can  evolve  and  remain  useful. 

These  three  "vehicles"  together  are  the  mechanism  through  vsfnich  an 
adaptive  design  approach  can  be  implemented.  They  ensure  a  systems 
design  environment  that  fosters  ready  communication  and  rapid 
implementation  of  lu lambiguously  communicated  desires  (user’s)  and 
intent  ions  (builder’s). 

Establishing  a  "Paradigm** 

To  design  a  proper  paradigm,  it  is  necessary  that  there  be  a  clear 
understanding  of  who  the  users  are,  and  what  their  areas  of  expertise 
(and  limitations)  are.  If  users  are  properly  identified,  a  paradigm 
that  provides  "procedures  and  data  representations  that  fit  well  with 
specific  managers’  established  activities"  can  be  designed  CMeador,  et 
al  :  1623.  3>iis  is  important  if  the  DSS  is  to  be  accepted.  A  support 
system  that  presents  a  non-technical  manager  with  statistical  summaries 
to  guide  his  decision  process  is  an  effective  way  of  ensuring  the  system 
is  not  used.  Likewise,  providing  a  system  that  1)  allows  the  manager  to 
view  data  pertaining  to  his  problem  (even  at  some  level  of  aggregation 
and  in  various  formats),  without  2)  providing  a  method  of  synthesizing 
its  impact  on  the  decision,  is  an  exercise  in  Management  Information 
Systems  (MIS),  not  DSS.  The  above  MIS  type  systems  are  designed  under 
vhat  Ackoff  calls  the  "give  them  more"  syndrome,  which  causes  managers 
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to  "suffer  ...  from  an  over  abundance  of  irrelevant  information  CAckoff 


:  147D.  A  proper  management  paradigm  will  make  a  management  system 
easily  used;  a  poor  paradigm  will  condemn  the  system  to  oblivion  because 
it  requires  too  much  of  the  user. 

Establishing  a  proper  paradigm  rests  on  the  identification  of 
users.  For  this  study,  the  users  to  be  supported  and  their  "general 
characteristics"  are  readily  definable  because  of  the  28  AD  air  crew 
management  structure;  identification  is  basically  a  matter  of 
observation  to  be  detailed  in  the  following  chapter. 

Factor  Identification 

Many  of  the  factors  that  bear  on  this  problem  were  identified  in  a 
previous  research  effort  in  this  area  CSchneiderl.  Some  were  addressed 
adequately,  some  need  more  detailing  before  they  can  be  useful  in  an 
implemented  system.  The  factors  that  received  the  most  attention  in  the 
previous  research  effort  were  those  that  dealt  with  manning  level. 

While  the  issue  of  experience  level  was  raised,  methods  for  determining 
these  levels  were  treated  scantily.  Identification  of  measures  of 
experience  and  methods  for  profiling  the  experience  base  of  the  crew 
force  were  not  adequate  to  guide  a  decision  process  attempting  to 
stabilize  experience  in  the  crew  force. 

The  factors  were  identified  in  discussion  with  air  crew  managers 
and  analysts  at  the  28  AD  during  the  course  of  multiple  PFT  conferences. 
In  most  cases,  the  28  AD  PFT  participants  indicated  that  they  were  able 
to  identify  factors  that  were  important  to  them,  and  that  they  attempted 
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to  estimate  their  levels  and  impact  on  the  AWACS  crew  force  in  making 
PFT  recommendations.  They  also  indicated  that  many  of  the  estimates 
were  indefensible,  as  they  were  generally  "best  guesses". 

liaming  Level  Factors.  For  the  purposes  of  this  research,  the 
factors  previously  identified  as  drivers  of  manning  level  were  accepted 
for  the  kernel  design.  Phone  interviews  with  AUIACS  managers  verified  no 
significant  changes  in  the  set  of  perceived  factors  have  occurred 
CQuzec].  These  factors  are  related  to  two  primary  areas:  1)  personnel 
losses  from  the  system  and  2)  changing  authorizations. 

The  personnel  loss  factors  are  derived  from  historical  databases 
and  are:  1)  the  time  spent  in  the  Air  Force,  and  2)  average  "tour" 
length.  The  first  factor  indicates  the  losses  AWACS  will  suffer  due  to 
separations  (end  of  enlistment,  retirement,  etc).  The  second  factor 
indicates  the  losses  due  to  Permanent  Change  of  Station  (PCS)  out  of  the 
AWACS  community. 

The  factors  that  bear  on  authorizations  for  each  crew  position 
were  discussed  briefly  in  chapter  one.  They  are  PAA,  crews  per  PAA, 
staff  authorizations,  and  number  of  personnel  per  crew.  These  factors 
are  set  outside  of  the  PFT  process,  being  part  of  overall  Air  Force 
structuring  policy.  As  such,  they  are  planned  well  in  advance  of  any 
PFT  planning  on  which  they  bear  and  are  matters  of  record. 

In  this  study,  none  of  the  above  factors  will  be  addressed  from 
the  perspective  of  utiy  they  occur.  Rather,  they  will  be  used  solely  for 
the  purpose  of  predicting  manning  shortfalls.  The  issue  of  (for 
example)  why  some  mid-career  flyers  choose  to  separate  before  retirement 
is  not  examined.  The  fact  that  some  percent  historically  do  separate 
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is  not  examined.  The  fact  that  some  percent  historically  do  separate 
early  is,  however,  of  use  in  predicting  acquisition  needs. 

Experience  Level  Factors.  Estimating  the  experience  level  of  a 
crew  position  entails  identifying  the  factors  that  can  describe 
experience,  building  a  model  to  estimate  the  individual  crew  member’s 
experience  (as  a  "score"),  and  then  using  the  estimated  score  to  profile 
the  experience  of  the  position.  This  profiling  can  be  done  in  any 
number  of  ways,  e.g.,  it  can  be  described  in  terms  of  mean  experience, 
median  experience,  or  via  a  graphical  presentation  of  the  experience 
distribution.  This  process  is  depicted  graphically  in  Figure  5. 

Unfortunately,  experience  level  quantification  is  difficult  to 
accomplish  for  most  of  the  mission  cre^v  positions.  This  is  because  no 
direct  measures  for  assessing  experience  of  the  individual  crew  members 
exist.  Estimates  of  individual  experience  must  be  based  on  surrogate 
measures,  where  the  choice  and  significance  of  these  measures  are 
unc 1 ear . 

The  first  step  in  assessing  the  experience  of  a  crew  position  is 
determining  what  surrogate  measures  are  useful  in  developing  "experience 
scores'  for  the  individual  crew  members  and  then  developing  the  scores. 
Examples  of  possible  surrogate  measures  are;  rank,  years  of  service. 
Standardization  Evaluation  (StanEval)  ratings,  number  of  flying  hours, 
number  of  missions  flown,  number  of  exercise  missions  flown,  etc. 

Since  appropriate  surrogate  are  uncertain  and  the  number  of  possible 
surrogates  needed  to  produce  an  "experience  score"  may  be  large,  any 
method  used  to  identify  factors  and  produce  scores  will  have  to  be 
capable  of  identifying  and  aggregating  multiple  factors  in  the  face  of 
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user  uncertainty.  Two  possible  methods  that  can  identify  the  necessary 
factors  and  directly  aggregate  them  into  usable  scores  are  considered. 


Figure  5  —  Experience  Profiling  Processs 


These  are  1)  Multi-Attribute  Decision  Analysis  (MADA),  and  2) 


regression.  These  methods  are  discussed  below. 


liADA  ffethods.  Estimating  experience  for  the  crew  members  can  be 
approached  from  the  perspective  of  "expert  Judgement".  There  are 
experts  in  the  AWACS  community  ubo  are  familiar  with  the  Jobs  each  crew 
member  performs  and  ubo  routinely  make  mission  decisions  based  on  their 
estimates  of  who  is  most  capable.  If  an  expert  judgement  approach  is 
used,  the  issue  becomes  one  of  eliciting  from  the  expert  the  uncertain 
factors  that  enter  his  decisions,  their  relative  weights,  and  the  trade¬ 
offs  (utility)  within  them.  Multi-Attribute  Decision  Analysis  is  an 
analysis  discipline  geared  to  structuring  problems  such  as  this,  where 
decisions  (answers)  are  plagued  by  uncertainty  CSmartEdge  Documentation 
:  B-2:. 

To  elicit  a  set  of  possible  factors  from  the  experts,  surveys  are 
used.  The  intent  of  the  surveys  is  to  establish  ranking  among  the 
possible  factors  (to  reduce  the  set  of  possible  factors  to  manageable 
proportions),  to  establish  that  the  expert  being  surveyed  is  consistent, 
and  to  look  for  possible  factors  that  have  not  been  considered.  Two  of 
the  standard  surveying  methodologies  for  developing  measures  on  1)  the 
ranking  of  factors  and  2)  respondent  (and  factor)  consistency  are  based 
on  semantic  scales  and  pair  comparison,  respectively  COian]. 

Solicitation  of  new  factors  to  be  considered  can  be  achieved  through 
simple  questionnaires. 

Appropriate  relative  weights  for  the  factors  and  the  utility 
function  that  describes  the  relationship  of  the  states  within  each 
factor  can  be  determined  through  a  series  of  reference  lotteries.  In  a 
"basic"  reference  gamble  procedure,  sequences  of  reference  gambles  are 
conducted  where,  prior  to  the  gamble,  payoffs  for  the  reference  gamble 
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are  established.  Then,  iteratively,  probabilities  (p)  for  winning  the 
gamble  are  assigned,  and  the  certainty  equivalent  (CD  for  the  gamble 
evaluated  (where  the  CE  is  the  point  uhere  the  expert  being  consulted 
indicates  he  is  indifferent  to  the  lottery  and  the  certain  value). 
Plotting  p  verses  CE  yields  the  utility  curve  for  the  factor  being 
examined  CHolloway  ;  4221. 

Modifications  of  the  basic  reference  are  available.  Holloway 
suggests  that  the  50-50  is  often  the  best  gamble,  since  people  often 
think  in  terms  of  go-no  go  choices  CHolloway  :  4281.  Here,  CEs  are 
first  determined  for  p  =  0,  .5,  and  1.  For  the  rest  of  the  procedure,  p 
is  fixed  at  .5  and  the  GEs  for  each  gamble  elicited.  The  probability 
for  the  certain  value  can  be  determined  from  the  gamble  conducted. 

The  lotteries  described  above  are  conducted  in  a  structured 
question  and  answer  session,  where  the  expert  indicates  his  preferences 
to  the  series  of  proposed  gambles.  This  lottery  structure  elicits  both 
the  factor  weights  and  the  utility  functions  that  underlie  the  expert's 
estimations.  Combination  of  the  weights  and  utilities  in  an  additive 
formulation  produces  a  "score"  for  individual  crew  members  uhen  the 
formula  is  evaluated  using  the  crew  member's  attributes  for  each  factor. 
(Reasonably  easy  to  use  commercial  software  packages  that  conduct  these 
types  of  question  and  answer  session  exist,  e.g.,  SmartEdge,  ver  (3)). 

Regression  Methods.  Two  classes  of  regression  treatments  suggest 
themselves  as  candidates  for  determining  appropriate  factors  for 
constructing  an  experience  score.  Both  require  that  the  crew  force  be 
"categorized"  in  some  manner  prior  to  regressing  to  identify  factors. 

The  "categories"  are  then  used  as  the  response  to  be  fit  by  a  regression 
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of  personnel  data.  The  set  of  regressors  that  are  significant  are  the 
factors  to  be  used  for  producing  experience  scores. 

Logistic  Regression.  Typically  this  type  of  regression  falls 
into  the  "legist ic  regression"  arena.  In  cases  uhere  the  response 
variable  is  categorical,  "theoretical  and  empirical  considerations 
suggest  ...  the  shape  of  the  response  function  will  frequently  be 
curvilinear  CNeter,  et  al  :  3613.  Logistic  transformations  of  the 
response  function  produce  responses  of  appropriate  shape  and  have  the 
desirable  property  of  being  asymptotic  to  the  extreme  values  of  the 
response,  i.e. ,  predicted  values  produced  using  the  logistic  response 
function  will  not  fall  outside  of  the  range  of  the  categories  of  the 
original  response  CNeter,  et  al  :  356-3631. 

Miltiple  Regression  with  a  Binary  Response.  If  the  initial 
categorization  of  the  crew  force  is  restricted  to  two  groupings,  (i.e., 
split  the  personnel  into  an  "expert  group"  and  a  "less  than  expert 
group"),  the  response  is  a  binary  response.  In  this  case,  a  multiple 
regression  is  feasible  CNeter,  et  al;  354-3613.  As  in  all  regression, 
the  factors  to  be  used  are  those  that  test  significant  in  predicting  the 
responses. 

The  methods  discussed  above  for  determining  appropriate  experience 
factors  and  producing  experience  scores  are  only  some  of  the  possible 
methods  for  dealing  with  the  problem  at  hand.  Certainly,  they  do  not 
constitute  the  entire  spectrum  of  options.  Each  has  problems  associated 
with  its  use.  In  some  cases,  the  problems  are  inherent  in  the  method 
itself  (regression  using  binary  categorical  responses).  In  others,  the 
problems  encountered  in  using  a  particular  method  are  related  to  the 
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environment  in  which  the  method  will  be  applied  (both  MADA  and  logistic 
regression  share  this  feature). 

In  the  following  chapters,  ^,>iich  deal  with  design  and 
implementation  of  the  DSS,  the  particular  problems  with  each  method  is 
discussed,  and  a  choice  of  method  for  the  kernel  design  is  selected  and 
explained. 
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III.  Overall  System  Design 


General 

Designing  for  adaptation  is  critical  for  the  decisions  being 
supported  by  this  research  effort.  The  decisions  to  be  made  go  beyond 
what  has  been  attempted  by  AWACS  air  crew  managers  in  the  past, 
explicitly  addressing  experience  level  inventories  and  the  need  to 
stabilize  both  manning  levels  and  experience  levels  in  the  various  AWACS 
crew  positions.  Since  the  decisions  being  supported  cannot  follow  an 
already  established  decision  process,  an  initial  process  must  be 
established  and  then  adapted  as  the  users  determine  exactly  uAiat  is 
required  to  fully  support  their  decisions. 

The  necessity  for  supporting  system  adaptation  is  also  inherent  in 
the  goals  of  the  decisions  being  supported.  The  goals  of  stabilizing 
manning  and  experience  in  the  AWACS  crew  positions  are  geared  to  change. 
Since  the  system  will  be  changing  the  environment  it  addresses,  it  must 
also  change  to  continue  to  address  the  environment  of  interest  and 
successfully  achieve  its  goals. 

The  specific  design  of  the  DSS  examined  in  this  research  effort,  is 
driven  by  two  major  concerns:  conceptual  design  considerations  that  are 
related  to  information  requirements  and  technical  considerations  that 
deal  with  how  the  system  is  implemented.  Both  of  these  "design 
elements"  must  be  addressed  in  the  adaptive  design  environment.  The 
conceptual  aspects  of  the  design  effort  deal  with  determining  user 
requirements,  identifying  a  decision  process  to  support  the 
requirements,  and  establishing  the  infrastructure  necessary  to  support 
the  system's  evolution.  The  technical  design  considerations  deal  with 
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providing  the  environment  uhere  the  decision  process  occurs. 

The  above  delineation  of  design  concerns  is  made  only  to  indicate 
that  there  are  two  dimensions  to  the  design  problem,  not  to  imply  that 
the  design  issues  can  be  treated  separately.  One  (the  conceptual 
portion)  is  primarily  the  realm  of  the  user  and  designer,  i^ile  the 
other  (the  technical  portion)  is  primarily  the  realm  of  the  builder. 
However,  there  is  not  a  clean  division  in  these  two  efforts.  F  jr 
example,  ti  e  act  of  specifying  that  the  system  be  able  to  adapt  has  an 
impact  on  the  technical  portion  of  the  design  effort,  and  requires 
choosing  specific  facilities  to  ensure  the  necessary  adaptations  to  the 
system  are  supported. 

The  following  sections  of  this  chapter  will  discuss  each  of  these 
design  areas.  While  the  areas  are  not  cleanly  divided  in  practice,  the 
discussion  of  each  area  will  be  seoarated  for  clarity. 

Conceptual  Design  Considerations 

The  concepti  v.l  design  issues  that  need  to  be  considered  in 
developing  a  DSS  can  best  be  addressed  frcm  the  context  of  two  elements 
of  the  adaptive  design  process.  These  elements  are  the  information 
requirements  determinations  (IRD)  phase  and  the  information  requirements 
analysis  (IRA)  phase.  During  the  "earlier"  IRD  phase,  user  requirements 
are  expressed  as  general  1 • ^ts  of  possible  requirements.  These 
"candidate"  requirements  are  subjected  to  scrutiny  during  the  IRA  phase. 
It  IS  here  that  the  actual  "succinct"  list  of  requirements  defining  the 
system’s  form  is  generated.  Each  of  these  phases  are  distinct  in  their 
goals.  However,  a  common  set  of  design  "tools"  can  be  used  for  each. 


and  are  those  that  were  discussed  under  the  methodology  of  adaptive 


design. 

The  conceptual  design  issues  in  this  research  were  addressed  in 
four  distinct  phases.  These  are:  1)  concept  mapping,  2)  storyboarding, 
3)  kernel  selection,  and  4)  establishment  of  a  paper  hook  book  to 
capture  the  road  map  for  future  development  of  the  system.  These  four 
phases  are  discussed  below. 

Concept  Mapping.  Initially,  a  concept  map  was  constructed  to 
describe  the  elements  of  the  problem  and  their  interactions.  The 
concept  map  was  generated  by  this  researcher  and  another  former  28  Air 
Division  analyst  CSumner].  (In  this  instance,  these  analysts  acted  as 
proxies  for  the  actual  decision  makers  to  be  supported).  Both 
participants  in  the  concept  map’s  generation  have  had  experience  with 
the  AUIACS  PFT  cycle,  and  have  participated  in  AWACS  PFT  conferences  in 
advisory  capacities. 

An  aggregated  concept  map  for  the  acquisition  problem  is  shown  in 
Figure  6.  Relationships  among  the  major  elements  are  easily  seen  in 
this  high  level  presentation. 

The  detailed  concept  map  for  the  overall  acquisition  problem  is 
included  in  Appendix  A.  The  map  is  by  no  means  complete;  its  function 
is  to  delineate  the  problem  space  sufficiently  to  allow  the  design 
process  to  continue  with  some  assurance  that  the  problem’s  total  scope 
has  beeri  considered. 

Storyboarding.  Storyboard  production  for  this  system  occurred  in 
three  cycles.  The  initial  storyboards  for  the  system  discussed  here 
were  conceptualized  at  the  28th  Air  Division  in  mid-1987  in  response  to 
dissatisfaction  with  aspects  of  the  storyboards  produced  by  Schneider 
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Figure  6  -  High  Level  Concept  Map  of  Acquisition  Problem 


CSchneiderl.  This  early  definition  of  the  desired  system  occurred 
during  brainstorming  sessions  among  the  28th  Air  Division  analysts  (Capt 
Geo.  Mark  Waltensperger ,  and  Lts  Kevin  M.  Holt  and  David  L.  Sumner). 

The  evolution  of  the  storyboards  continued  at  AFIT  vrfnen  Holt  and 
Sumner  were  formally  exposed  to  DSSs  and  storyboarding.  Here,  the  AWACS 
PFT  problem  was  used  as  a  vehicle  to  better  understand  the  structures 
and  goals  of  storyboarding.  In  the  process  of  developing  an 
understanding  of  storyboarding,  a  system  that  would  support  28  AD 
personnel  acquisition  was  captured  in  a  "complete"  set  of  storyboards. 
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Finally,  the  process  of  this  research  led  to  modifications  of  the 
storyboards  as  the  problem  was  addressed  in  more  detail.  The  final 
storyboards  were  evaluated  by  Sumner,  acting  as  a  proxy  decision  maker, 
to  ensure  that  they  continued  to  define  the  form  of  the  desired  system 
and  to  ensure  that  the  system  addressed  "all"  of  the  user’s  decision 
support  needs.  Three  of  the  storyboards  are  shown  below,  and  the  ROMC 
(as  discussed  in  chapter  21  of  each  is  detailed.  The  complete  set  of 
"final"  storyboards  are  included  as  Appendix  B. 

Representative  Storyboards.  Figure  7  is  the  home  display  for 
the  system  when  it  is  first  entered.  (As  such,  it  is  also  the  kernel  of 
the  DSS,  which  is  discussed  in  detail  in  the  next  chapter).  The  user 
enters  the  system  to  a  display  that  graphically  represents  the  current 
experience  profile  of  the  crew  force  he  manages. 

Here,  the  state  of  the  crew  position  is  shown  by  the  distribution 
portrayed  in  histograms.  The  left  histogram  indicates  the  impact  of 
known  losses  (i.e. ,  known  retirements,  known  PCSs,  etc.)  on  the 
position’s  experience  while  the  right  indicates  a  best  case  for 
experience  based  on  recapturing  all  known  "eligible"  ex-AWACS  personnel 
in  the  Air  Force  manning  pool.  (The  top  portion  of  the  stacked  bars 
indicate  experience  contributions  by  members  being  lost  or  potentially 
recaptured) . 

Operations  are  minimal  for  this  screen.  At  the  decision  command 
level  (top  menu),  the  user  has  the  option  of  changing  the  crew  position 
being  profiled  and  of  selecting  different  manning  profiles.  At  the 
systems  support  command  level  (bottom  menu),  he  can  select  any  option. 


34 


Memory  aids  are  embodied  in  three  separate  areas  of  the  screen. 
First,  the  top  line  of  the  screen  indicates  to  the  user  specifically 
what  he  is  looking  at.  Second,  the  command  bar  has  the  applicable 
command  that  generates  the  representation  highlighted  (and  also 
indicates  decision  commands  that  can  be  executed  during  the  current 
operation,  i.e. ,  change  the  crew  position  to  be  examined).  Last,  the 
coding  of  the  bar  segments  is  documented  below  the  histograms. 
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Finally,  the  control  mechanism  of  the  system  is  embodied  in  the 
command  selection  procedures  and  options  available  to  the  user.  In  this 
instance,  the  user  can  select  «iiy  of  the  system  support  options  from  the 
bottom  command  bar  (and  these  commands  are  purposely  separated  from  the 
commands  that  bear  on  the  decision  process),  or  he  can  examine  other 
manning  profiles  that  are  of  interest  to  him. 

Figure  0  is  the  display  where,  ultimately,  the  final  decision  is 
made  about  which  acquisition  strategy  is  best.  Manning  is  not  an 


Figuve  8  -  (inabilities  Representation  of  the  DBS 
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end  to  itself;  a  mission  effective  AUIAC  capability  is  the  goal,  and  this 
goal  is  manning  acquisition  is  an  issue. 

For  each  of  the  acquisition  strategies  the  manager  considers,  he 
can  examine  the  manning  consequences  on  AWACS  mission  capabilities. 
Capabilities  information  is  represented  three  ways:  graphically  by  a 
line  that  is  plotted  relative  to  reference  values,  numerically  at  a 
detailed  level  in  a  table  below  the  graph,  and,  finally,  in  a  table  that 
shows  crew  positions  in  the  order  in  uiiich  they  limit  capabilities  the 
most . 

Operations  include  the  ability  to  "zoom"  to  each  crew  position  to 
examine  its  specific  impact  on  capabilities  and  accessing  system  support 
functions  such  as  getting  printouts  of  the  representation  or  capturing 
thoughts  in  the  hookbook  or  notepad.  ("Zooming"  is  accomplished  by 
selecting  the  appropriate  number  from  the  capabilities  limitation  list). 
Memory  aids  and  control  mechanisms  are  largely  the  same  as  those 
discussed  in  Figure  7. 

Figure  9  is  the  storyboard  for  the  DSS  function  of  keeping  the 
user  abreast  of  scheduled  changes  in  the  manning  structure  of  the  AWACS. 
It  is  a  tabularly  oriented  representation  of  the  changes  in  the  four 
factors  that  cause  manning  requirements  at  the  systems  level.  The  user 
is  shown  the  current  values  for  each  of  the  factors  in  the  BASE  column, 
and  is  shown  when  and  by  how  much  (relative  to  the  current  values) 
changes  in  factors  will  impact  each  crew  position. 

Operations  are  restricted  to  causing  the  crew  positions  and  their 
relevant  information  to  scroll  in  their  windows  and  to  accessing  system 
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Figure  9  -  Force  Structure  Representation  of  the  DSS 


and  decision  commands.  Memory  aids  and  control  mechanisms  remain  the 
same  as  in  figure  7. 

These  three  storyboards  are  representation  oriented,  having 
minimal  operations  input  from  the  user.  A  heavily  operations  oriented 
figure  is  discussed  later  in  this  chapter. 

Storyboarding  Contributions  to  Uhdarst ending  System  Needs.  As 
indicated  in  the  previous  chapter,  both  the  concept  map  and  storyboards 
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inipact  the  selection  of  the  kernel  system  to  be  developed.  The 
storyboard  process  immediately  reinforced  this  fact.  This  process 
reduced  the  scope  of  u4iat  was  addressed  to  a  small  portion  of  the 
concept  map,  choosing  not  to  directly  address  why  perturbations  (from 
desired  stable  conditions)  of  AWACS  manning  occurred,  but  rather 
focusing  on  how  to  correct  perturbations  when  they  occur.  The 
storyboards  focus  on  a  system  that  allows  managers  to  identify  manning 
deficiencies,  examine  means  of  correcting  the  deficiencies,  and  portrays 
the  consequences  on  manning  structure  of  whatever  acquisition  decisions 
are  made. 

It  was  also  during  the  process  of  storyboarding  that  the  issue  of 
exactly  "uho"  the  users  being  supported  were  was  addressed  in  detail. 
Until  this  point,  the  problem  was  looked  at  largely  from  the  perspective 
of  stabilizing  the  crew  force  in  a  prescriptive  manner,  not  from  the 
perspective  of  a  decision  process.  User  needs  and  impact  on  the  system 
had  not  been  considered.  The  specific  characteristics  of  the  users  were 
addressed  at  this  point,  and  the  "shape"  of  the  DBS  that  could  support 
them  determined.  Consequently,  storyboarding  was  critical  to  guiding 
the  formulation  of  the  decision  process  to  be  implemented  in  the  initial 
system  and  the  paradigm  through  which  the  decision  cycle  would  be 
expressed.  The  issues  of  decision  process  and  paradigm  are  discussed 
next. 

The  Decision  Process.  As  stated  above,  this  system 
addresses  areas  in  AI^JACS  crew  management  not  addressed  previously. 
Lacking  a  process  to  model,  an  initial  process  to  support  the  user's 
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decisions  had  to  be  developed.  This  decision  process  could  then  be 
modified  <or  totally  new  processes  implemented  so  that  the  problem  can 
be  addressed  multiple  ways)  as  the  users  determined  new  needs  (or 
desires)  for  support  in  addressing  personnel  acquisition  decisions. 

The  issue  of  supporting  particular  decision  processes  is  widely 
discussed  in  the  DSS  literature.  It  is  generally  held  that,  to  provide 
effective  support,  a  DSS  must  not  constrain  decision  makers  to  any  one 
view  of  how  to  approach  a  decision.  The  user  should  be  allowed  to 
establish  his  own  "best"  way  to  address  the  problem  being  supported 
CRobey,  et  al ;  Cohenl.  In  the  initial  stages  of  the  design  of  this  DSS, 
however,  some  process,  not  necessarily  the  "best",  was  required  as  a 
starting  point  from  which  growth  and  adaptation  could  proceed.  A 
decision  process  was  therefore  established. 

The  basic  decision  cycle  established  by  the  storyboards  is  to 
identify  the  current  state  of  the  crew  position  with  respect  to 
experience  and  manning  level,  and  then  proceed  to  a  "vhat-if"  cycle  to 
examine  various  decision  alternatives.  During  the  ”uhat-if"  cycle,  a 
manager  can  assess  multiple  strategies  for  correcting  crew  deficiencies 
based  on  a  clear  understanding  of  the  crew's  current  state.  Support  is 
also  provided  for  direct  examination  of  the  personnel  databases  and  for 
examining  scheduled  changes  in  the  AWACS  force  structure  that  will  alter 
the  manning  state. 

The  Decision  Paradigm.  Typically,  the  managers  to  be 
supported  by  the  proposed  DSS  are  senior  crew  members  assigned  to  a 
Tinker  AFB  AMACS  unit.  Uhiformly,  the  managers  assume  the  Job  of 
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managing  their  respective  crew  positions  as  a  duty  in  addition  to  their 
normal  flying  responsibilities.  To  support  these  managers  effectively, 
system  design  must  occur  with  the  understanding  that  crew  management  is 
only  one  facet  of  the  managers  Job,  and  not  necessarily  the  most 
important  (in  the  managers  eyes).  First-and-foremost ,  it  must  be 
recognized  that  the  managers  are  crew  members,  not  personnel 
special ists. 

The  management  paradigm  for  this  system  must  reflect  the  user’s 
capabilities  and  allow  him  to  make  decisions  without  being  burdened  by 
the  system.  A  proper  management  paradigm  makes  the  management  system 
easily  used;  a  poor  paradigm  will  condemn  the  system  to  oblivion  because 
it  requires  too  much  of  the  manager. 

To  ensure  the  system  is  useful  to  the  identified  user,  a  simple 
graphical  paradigm  was  adopted.  The  estimated  state  of  the  crew 
position  is  depicted  as  a  line  that  can  be  compared  to  a  line 
representing  the  desired  crew  state.  (It  is  possible  that  as  managers 
become  familiar  with  the  system  and  gain  expertise  in  making  decisions 
with  its  aid,  they  may  find  statistical  summaries  useful,  particularly 
when  trying  to  defend  their  need  estimates  to  higher  headquarters.  This 
can  be  added  to  the  design  at  a  later  date.  Initially,  however,  the 
system  should  be  limited  to  graphical  portrayals  of  decision  results. 
These  are  easily  understood  and  tend  to  limit  the  manager’s  sense  of 
information  overload). 

In  Figure  10,  the  dashed  line  represents  the  desired  manning  level 
for  the  crew  position,  and  is  specified  by  authorization  levels  mandated 
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by  force  structure  considerations  (show  in  Figure  9).  The  solid  line  is 


Figure  10  -  Scheduling  Representation  of  the  DSS 


the  projected  manning  level  based  on  current  acquisition  decision 
parameters.  The  manager’s  objective  is  to  make  the  solid  line  match  the 
dashed  line  by  suitable  choice  of  values  for  the  acquisition  decision 
parameters.  The  manning  level  projections  that  arise  due  to  the  manager 
changing  parameters  are  graphed  as  the  dotted  line,  and  can  be  compared 
to  the  desired  levels  and  to  the  levels  that  result  from  current 
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parameter  settings.  (Relative  to  the  previously  discussed  storyboards, 


this  screen  is  uftere  the  user's  major  operations  occur.  He  can  change 
multiple  parameter  values,  and  save  intermediate  combinations  of 
settings  for  comparison  of  the  results  of  his  various  "what-if" 
examinations  of  acquisition  possibilities). 

Kernel  Selection.  Uhile  the  concept  map  addresses  the  overall 
problem  space  and  the  storyboards  restrict  the  space  to  a  subset  the 
users  want  supported,  the  overall  system  specified  is  still  a  major 
undertaking.  To  attempt  to  give  the  users  a  tool  that  allows  them  to 
start  addressing  their  problem,  it  was  decided  that  an  appropriate 
kernel  would  be  the  capability  to  assess  current  experience  in  the 
various  crew  positions.  The  kernel's  selection  and  development  is 
discussed  fully  in  the  next  chapter. 

Continued  Adaptation.  The  mechanisms  for  fostering  continued 
system  adaptation  are  embodied  in  two  capabilities  included  in  the 
system.  These  are  the  hook  book  discussed  previously,  and  an  explicit 
assumption  review  capability  triggered  by  the  "ASSUMPTIONS"  button 
described  in  the  storyboards. 

The  hook  book  was  started  in  a  paper  form  at  the  beginning  of  the 
design  process  to  capture  needs  for  system  evolution.  (These  hook  book 
entries  are  included  in  Appendix  C).  This  was  important  to  the  system's 
development  in  that  it  captured  evolutionary  needs  without  squandering 
design  resources  on  redefinition  of  the  problem.  In  other  words,  it 
allowed  kernel  development  to  continue,  rather  than  causing  the  kernel 
to  be  continually  redefined.  In  its  system’s  integrated  form,  the  hook 
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bcxik  serves  the  same  function,  allowing  the  user  to  capture  needs 
without  unduly  interrupting  the  decision  cycle  in  Vstiich  he  is  engaged. 

The  explicit  inclusion  of  an  "ASSUMPTION"  button  was  modeled  after 
that  described  by  El  Sherif,  et.  al,  in  their  discussions  of  support 
systems  developed  for  the  Egyptian  cabinet  CEl  Sherif,  et.  al .  :  5603. 

It  was  included  as  an  adaptive  design  mechanism  because  it  forces  the 
user  into  an  awareness  that  decision  making  depends  on  more  than  the 
data  that  is  manipulated.  How  and  why  the  data  are  manipulated,  and  the 
meaning  of  the  results  can  be  very  assumption  sensitive.  If  the  system 
is  to  adapt  successfully,  any  assumptions  that  are  inherent  in  the 
system's  implementation  must  be  open  to  scrutiny  and  change. 

Having  discussed  the  adaptive  design  elements  of  this  research, 
the  remainder  of  this  chapter  is  devoted  to  a  discussion  of  the 
technical  design  considerations  of  the  DSS  considered  in  this  research. 

Technical  Desicn  Considerations 

The  technical  design  considerations  for  this  effort  are  as 
extensive  as  those  of  adaptive  design,  and  are  equally  as  important. 
Technical  elements  of  a  system's  design  can  relegate  a  conceptually 
elegant  system  to  a  less-than-sterl ing  system  when  implemented,  with 
incorrect  selection  of  technical  design  components  hindering  system 
development  and  use.  Technical  design  considerations  include: 

1)  choosing  the  particular  hardware  and  software  environment  to  be 

used; 

2)  identifying  and  planning  access  to  data  necessary  to  support 
the  decision  process; 

3)  identifying  places  in  the  decision  process  where  models  are 
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necessary  (or  desirable)  to  support  decisions,  and 

4)  choosing  appropriate  model  formulations  and  managing  the  "model 
base". 

Environment.  Choosing  the  hardware  environment  bo  be  used  was  a 
non-issue  for  this  DSS.  The  28  Air  Division  has  invested  heavily  in 
Zenith  Z-240S,  with  networking  of  the  systems  in  a  beginning  stage.  All 
air  crew  managers  have  ready  access  to  these  machines. 

The  software  environment  is  not  as  clear  cut,  other  than  the 
requirement  that  the  software  run  on  Z-248  cofrputers.  Ideally,  a  DSS 
generator  would  be  employed  to  design  and  implement  the  desired  system. 
However,  this  option  was  not  examined  for  two  reasons.  First, 
procurement  of  the  software  in  time  for  use  in  this  effort  presented 
problems.  Second,  it  was  beyond  the  scope  of  this  effort  to  even 
identify  an  effective  DSS  generator  (if  it  existed)  that  could  support 
the  requirements  entoodied  in  the  storyboards.  (In  another  Air  Force  DSS 
development  effort,  two  years  were  spent  evaluating  numerous  software 
packages  "claiming"  to  be  DSS  generators  CWalker,  et  all.  A  large 
number  of  candidates  were  examined,  and  more  than  hal f  were  el iminated 
from  the  evaluation  process,  because  they  were  clearly  not  DSS 
generators,  regardless  of  the  vendor's  claims'). 

Another  environment  rejected  for  the  implementation  of  this  DSS 
was  examined  in  the  previous  research  effort  in  this  area.  Schneider 
examined  the  feasibility  of  using  an  off-the-shelf  software  package, 
"□vmi",  as  the  environment  for  generating  a  DSS.  The  windowing 
abilities  and  the  integrated  database,  spreadsheet,  graphics  and  word 
processing  features  in  this  package  were  exercised  extensively  in 


developing  a  prototype  DSS,  and  "ENABLE"  was  found  adequate  to  produce  a 
workable  DSS. 

From  the  user's  perspective,  however,  this  environment  had  severe 
shortcomings  that  would  preclude  its  use  as  the  environment  of  final 
implementation.  The  test  system  that  was  demonstrated  at  the  28  AD  was 
driven  with  small  test  databases  and  simplified  models.  It  was,  none- 
the-less,  incredibly  slow,  vhich  aggravated  the  operators,  and  it  was 
uncomfortable  to  operate  because  of  the  way  "ENABLE"  transitions  between 
its  various  "integrated"  modules  (i.e. ,  screen  bounce  and  flash,  and 
"ENABLE"  environment  menus  flickering  past  as  the  package  made 
transit  ions) . 

In  general,  the  man  machine  interface  can  be  a  major  factor  in  a 
system's  success,  and  speed  is  a  major  component  of  this  interface. 

Slow  systems  are  perceived  as  being  "unresponsive"  to  the  user,  and  are 
therefore  avoided.  For  "ENABLE”  (and  probably  any  integrated  "office" 
software),  speed  was  a  compound  issue.  In  the  integrated  environment, 
trade-offs  are  made  in  the  capabilities  of  the  various  modules,  with  no 
module  being  the  equal  of  the  best  stand-alone  packages  that  are 
available  to  perform  the  same  function.  Further,  the  very  fact  of 
having  an  integrated  environment  in  a  general  package  entails  overhead 
to  exercise  all  of  the  package's  abilities,  including  those  that  are  not 
needed  for  a  particular  application  (say,  a  DSS).  This  is  probably  the 
case  for  any  general  purpose  system;  they  will  always  have  overhead  to 
support  requirements  not  necessary  for  a  DSS,  and  the  DSS  suffers. 

For  the  reasons  alluded  to  above,  the  final  environment  for 
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implementing  the  DSS  examined  here  should  be  provided  by  a  stand-alone 
program  that  provides  the  user  interface  prescribed  by  the  storyboards, 
which  executes  all  of  the  desired  functions,  and  t«^ich  supports  other 
functions  only  as  their  needs  are  identified  during  the  adaptive  design 
process.  Unfortunately,  this  has  undesirable  consequences  for  systems 
adaptations.  These  consequences  are  discussed  in  Chapter  5. 

Data.  The  data  to  support  this  DSS  is  available  at  the  28  AD. 
However,  it  is  not  in  a  form  that  can  be  directly  used  by  any  automated 
system.  (A  large  part  of  the  data  is  in  word  processing  documents  that 
are  updated  daily). 

An  AWACS  wide,  standardized  database  for  crew  management  functions 
that  contains  most  of  the  necessary  data  has  been  designed  using  DBASE 
III.  When  it  is  in  place,  implementation  of  this  DSS  can  begin.  The 
other  necessary  data  elements  that  are  not  part  of  the  above  database 
can  be  incorporated  into  a  small  database,  or  are  on  base  level 
computers  from  which  they  can  periodically  be  down-loaded  in  Z-248 
readable  formats  and  converted  to  DBASE  III  file  formats  for  use  in  the 
DSS.  (The  necessary  data  elements  and  their  sources  are  included  in 
Appendix  D). 

McxJels.  This  DSS  requires  three  major  model  types  to  synthesize 
personnel  data  and  drive  the  displays  providing  the  decision  testing 
mechanism.  First,  models  to  project  losses  in  each  crew  position  are 
necessary.  These  models  will  be  used  to  provide  the  loss  estimates  to  a 
simple  additive  model  that  drives  the  manning  level  displays,  and  will 
also  feed  the  models  driving  the  experience  profile  displays.  (These 
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are  the  displays  shown  in  Figures  10  and  7  respectively).  Since  these 
models  tie  into  experience  level  projections,  they  will  necessarily  be 
complex.  They  will  have  to  project  losses  in  distinct  categories  in 
each  crew  position,  where  the  categories  are  defined  by  a  personnel 
factor  that  is  important  in  estimating  experience. 

Second,  models  for  each  crew  position  that  assign  "experience 
scores"  to  the  members  of  the  crew  position  are  needed.  These  scores 
will  provide  the  data  used  to  profile  the  experience  in  the  position. 
(These  scores  are  aggregated  to  produce  the  histograms  shown  in  Figure 
7). 

Finally,  a  model  that  evaluates  AWACS  mission  capability  as  a 
function  of  manning  and  experience  levels  is  required.  (This  is  the 
model  that  drives  the  display  shown  in  Figure  9). 

Each  of  these  models  will  require  extensive  management  in  that 
they  require  continual  updating  crttil  such  time  as  the  crew  positions 
they  address  become  stable  (and  periodic  examination  there  after). 

Model  management  will  entail  AWACS  analysts  periodically  rebuilding  the 
models.  Model  accuracy  will  change  as  factors  pertinent  to  the  model's 
function  change,  due  to  outside  forces  (e.g.,  force  reduction  that 
remove  classes  of  people  in  an  unexpected  manner),  or  due  to  actions 
resulting  from  the  use  of  the  DSS  that  alter  the  force  structure. 

The  models  that  drive  the  system  are  the  major  "repository"  of 
assumptions  that  the  system  functions  under.  Therefore,  in  addition  to 
updating  the  models,  the  analysts  have  a  critical  responsibility  to 
document  the  changes  in  assumptions  that  are  inherent  in  changing  the 
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mcxlels.  (As  stated  previously,  review  of  assumptions  is  critical  if  the 
system  is  to  remain  useful).  More  information  on  models  can  be  found  in 
Appendix  E. 

Having  discussed  the  overall  system  design,  the  following  chapter 
deals  with  the  design  of  the  kernel  DSS. 
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IV.  Kernel  Design 


The  overall  system  discussed  in  the  previous  chapter  will  entail  a 
large  and  time  consuming  effort  to  implement  in  its  entirety. 

Therefore,  some  portion  of  the  system  needed  to  be  addressed  in  detail 
to:  1)  demonstrate  to  the  user  that  progress  was  being  made,  2)  to  help 

the  user  solve  some  portion  of  his  problem,  and  3)  to  put  an  initial 
tool  into  the  user’s  hands  so  that  the  tool's  use  would  help  the  user 
refine  his  perceptions  of  his  needs.  To  get  the  system  up  and  running, 
a  kernel  was  selected  that  provides  managers  an  initial  capability  in 
assessing  acquisition  needs  vhere  they  currently  had  none. 

Kernel  Selection 

Managers  currently  deal  with  acquisition  planning  solely  from  a 
short  term  "nuntoers"  perspective.  They  plan  personnel  acquisitions  to 
correct  any  shortfall  from  authorized  manning  levels  that  they  see 
arising  over  the  near  term  life  of  the  crew  force.  Even  though 
acquisitions  are  managed  to  correct  short  falls  in  manning  level,  there 
is  a  widespread  recognition  in  the  AWACS  community  that  experience  in 
the  crew  force  should  also  be  managed. 

Because  of  the  difficulties  in  determining  experience,  experience 
profiles  of  the  crew  positions  have  been  measured  against  arbitrarily 
specified  references,  (i.e. ,  all  WDs  having  one  year’s  flying  experience 
are  "experienced"  and  all  CTs  with  SOO  E-3  hours  are  "experienced".  In 
general,  from  the  author’s  experience  at  the  2aAD,  these  measures  of 
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experience  are  viewed  with  varied  levels  of  disbelief). 

These  references  are  so  broad  as  to  be  meaningless.  The  only 
people  that  they  exclude  from  the  "fully  experienced"  category  are  those 
recently  graduated  from  primary  crew  training  at  the  552  TTS. 

Therefore,  the  kernel  DSS  that  was  designed  was  developed  to  address  the 
crew  experience  issue.  Specifically,  the  kernel  system  allows  managers 
to  graphically  profile  current  experience  of  crew  positions  with  a  fair 
degree  of  detail,  rather  than  examining  them  in  the  current  arbitrary, 
binary  fashion.  (At  some  future  time,  a  referent  curve  of  minimum 
expertise  levels  in  each  experience  category  should  be  developed  to 
allow  the  manager  to  assess  how  close  he  is  to  a  tolerable  manning 
profile.  Uhtil  that  time,  managers  will  have  to  make  determinations  on 
the  desirability  of  the  experience  profile  from  the  gross  pattern 
evidenced  in  the  graphical  display). 

Choosing  a  kernel  DSS  to  address  current  experience  in  a  crew 
position  makes  good  sense  for  many  reasons.  These  reasons  are 
introduced  here,  and  discussed  in  the  following  section  of  this  chapter. 
The  first  reason  selecting  the  above  kernel  is  that  it  gives  managers  a 
capability  that  they  want.  Second,  even  though  it  does  not  directly 
address  the  need  to  correct  the  current  short  term  management  approach 
to  the  acquisition  process,  actions  taken  as  a  result  of  decisions  based 
on  experience  level  estimates  will  not  aggravate  manning  level 

conditions.  Third,  it  allows  managers  to  use  an  experience  profiling 
tool  and  refine  it  before  work  is  expended  in  building  the  more  complex 
profiling  tool  needed  to  project  experience  levels  in  future  years. 
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Lastly,  it  is  one  area  of  the  acquisition  planning  process  utiere  a 
concerted  effort  is  needed  to  achieve  any  results.  The  value  of  giving 
users  something  they  desire  is  self  evident.  The  other  three  issues  are 
discussed  next. 

Rationale  for  the  Kernel  Selection.  The  capability  to  examine 
current  experience  levels  allows  managers  to  approach  the  acquisition 
process  with  an  ability  to  address  needs  that  he  cannot  currently 
assess.  With  this  capability,  he  can  take  action  to  correct  experience 
deficiencies  by  altering  the  types  of  people  acquired.  This  augments 
his  management  capabilities  without  altering  his  current  management 
scheme,  <^ich  only  addresses  identifying  the  number  of  people  to  be 
acquired.  Since  the  manager  is  not  changing  the  manner  in  which  he 
decides  how  many  accessions  are  needed,  experience  profiling  cannot 
aggravate  AWACS  manning  relative  to  the  current  management  scheme.  The 
current  acquisition  procedure  is  simply  not  affected  by  it. 

The  net  result  of  the  kernel  DSS  is  an  ability  for  the  manager  to 
identify  times  when  AFMPC  must  make  some  of  the  annual  crew  acquisition 
come  from  sources  with  either  1)  AWACS  experience  (i.e. ,  allow  AWACS  to 
recapture  previous  AWACS  resources),  or  with  2)  transferable  experience 
(e.g.,  pull  computer  technicians  into  the  CDMT  position,  rather  than 
bringing  in  brand  new  Air  Force  acquisitions).  These  conditions  can 
occur  when  the  current  aggregate  experience  of  the  crew  position  is  low 
and  the  addition  of  totally  inexperienced  personnel  to  the  position  will 
only  aggravate  the  condition.  Therefore,  the  effect  of  the  kernel  DSS 
will  be  to  elevate  the  experience  level  of  the  crew  force  for  some 
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extended  period  of  time.  (Even  if  elevating  the  experience  level  of  the 
crew  force  were  not  necessary,  it  would  certainly  not  be  damaging  to 
have  a  crew  force  "overly  rich"  in  experience). 

Referencing  the  point  about  letting  managers  use  a  "simple" 
experience  profiling  tool,  there  are  two  reasons  for  implementing  the 
chosen  kernel.  In  the  first  place,  the  basic  work  required  for  the  long 
term  experience  profiling  is  embedded  in  the  kernel  system.  The  basic 
issue  of  how  to  build  an  experience  "score"  is  addressed  here. 

Second,  projecting  profiles  requires  the  ability  to  accurately  project 
manning  losses  and  the  development  of  an  apportionment  scheme  for  any 
factors  that  impact  experience,  e.g.,  if  flying  hours  are  a  significant 
factor  in  determining  experience,  all  projected  AWACS  flying  hour 
allocations  must  be  distributed  over  the  crew  force  in  some  manner  to 
project  how  the  individuals  in  it  gain  experience.  Resolving  these 
issues  will  entail  a  significant  effort.  Before  the  effort  is  expended, 
the  basic  experience  profiling  capability  should  be  used  extensively  to: 
1)  familiarize  managers  with  experience  profiling,  and  2)  refine  the 
experience  scoring  methods  if  managers  feel  it  is  necessary. 

Finally,  experience  is  an  acquisition  issue  requiring  a  concerted 
effort  if  the  manager's  capabilities  to  address  the  issue  are  to  be 
improved.  While  the  crew  managers  currently  deal  with  acquisitions  on  a 
near-term  manning  level  basis,  improvements  in  AWACS  manning  stability 
can  be  achieved  with  relatively  modest  effort.  As  discussed  in  chapter 
two,  fluctuations  in  AUiACS  manning  are  cyclic  in  nature.  Recognizing 
this,  if  the  past  annual  final  PFT  acquisitions  are  examined  to 
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determine  where  peak  acquisition  demands  occur,  the  cycles  can  be  damped 
by  over-acquiring  personnel  in  the  period  Just  before  these  peaks.  In 
contrast,  there  is  no  usable  information  on  experience,  since  experience 
has  been  ignored  in  past  acquisition  management.  Any  attempt  to 
directly  deal  with  experience  required  a  "start  from  scratch".  The 
kernel  DSS  provides  that  start. 

With  the  above  understanding  as  to  why  the  particular  kernel  DSS 
was  chosen,  the  remainder  of  this  chapter  discusses  the  modeling  methods 
examined  to  develop  experience  scores  and  a  test  implementation  using  a 
spreadsheet  environment  to  examine  the  aggregated  experience  scores. 

Experience  Score  Modeling  Examination 

Three  different  modeling  approaches  w«re  examined  for  developing 
experience  scores  for  the  individuals  in  each  crew  position.  These 
were;  1)  logistic  regression  with  more  than  two  categorical  responses 
(10  in  this  case),  2)  MCDM  techniques  that  elicit  and  quantify  experts 
opinions  to  produce  "scores",  and  3)  multiple  regression  with  a  binary 
categorical  response.  Each  is  discussed  below. 

Lx)gistic  Regression.  Logistic  regression  approaches  to  developing 
experience  scores  suggest  themselves  because,  as  stated  in  chapter  2, 
empirical  evidence  suggests  that  response  functions  developed  using 
categorical  dependent  variables  will  frequently  be  curvilinear. 

Logistic  transformations  of  the  response  function  produce  responses  of 
appropriate  shape  and  have  the  desirable  property  of  being  asymptotic  to 
the  extreme  values  of  the  response,  i.e. ,  predicted  values  produced 
using  the  logistic  response  function  will  not  fall  outside  of  the  range 


54 


of  the  categories  of  the  original  response  CNeter,  et  al  :  356-363D. 

Taking  a  logistic  regression  approach  to  this  problem  requires 
that  the  current  crew  force  be  separated  into  distinct  categories  of 
experience.  Therefore,  this  approach  would  not  be  purely  a  regression 
approach,  but  would  require  some  preliminary  work  to  categorize  the 
members  of  each  crew  position.  Multi-Attribute  Decision  Analysis  (MADA) 
techniques  could  be  used  to  get  "expert  evaluations"  of  the  various  crew 
members. 

Unfortunately  for  the  current  problem,  logistic  regression 
requires  large  samples  at  each  of  the  possible  combinations  of  input 
factors  to  weight  the  regression  appropriately  CIMeter,  et  al  :  3643. 

For  the  current  problem,  the  largest  crew  force  is  in  the  250  person 
range.  If  only  four  predictors  at  two  levels  are  used,  the  largest 
sample  for  a  given  cofrtbination  of  inputs  is  only  15,  which  is 
insufficient.  (Given  the  current  perceptions  among  AUIACS  experts  as  to 
what  factors  may  be  useful  for  determining  experience,  it  is  unlikely 
that  a  reasonable  regression  can  be  achieved  without  more  than  four 
predictors  constrained  to  two  levels). 

Another  problem  with  this  approach  is  related  to  the  expert 
support  required  to  do  the  initial  categorization  of  the  crew  members  in 
the  various  crew  positions.  Discussion  of  this  problem  is  deferred  to 
the  next  section,  where  MCDM  methods  are  discussed. 

These  difficulties  with  the  logistic  regression  approach  removed 
its  use  from  consideration. 
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new.  Multi-Criteria  Decision  Making  techniques  suggest 
themselves  as  possible  candidates  for  addressing  this  problem  because  of 
the  lack  of  definitive  experience  measures  for  AUIACS  crew  members. 
Lacking  suitable  measures,  the  problem  can  be  approached  from  the 
perspective  of  expert  opinion.  MCDM  techniques,  particularly  Multi- 
Attribute  Decision  Analysis  (MADA),  are  ideally  suited  for  this  type  of 
approach.  Therefore,  MADA  approaches  to  this  problem  were  examined. 

The  details  of  this  examination  can  be  found  in  Appendix  F.  The  major 
conclusions  of  the  examination  are  repeated  here. 

The  MADA  methodology  examined  gave  results  that  made  sense. 

However,  uhile  the  factor  set  that  was  examined  was  quite  small,  the 
techniques  still  consumed  a  significant  block  of  an  expert's  time. 
Further,  v^en  multiple  experts  differ  on  utilities  and  weights,  MADA  has 
some  problems.  Various  concordance  measures  are  available  for 
determining  whether  differences  observed  between  the  elicited  values  are 
significant.  Uh fortunately,  uhen  the  concordance  measures  indicate 
significant  disagreement,  MADA  currently  has  no  generally  accepted  means 
for  resolving  the  dispute.  These  differences  in  expert  opinion  are 
bound  to  arise  v^en  these  techniques  are  applied  to  the  DSS  considered 
here.  How  they  are  resolved  is  critical  if  these  techniques  are  used. 

The  most  significant  problem  for  this  particular  application  was 
the  time  consumption  mentioned  above.  The  level  of  expert  involvement 
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required  to  implement  this  type  of  strategy  is  enormous.  For  the 
examination  of  M<®A,  weights  and  utilities  were  only  developed  for  two 
factors.  The  process  still  took  over  six  hours  to  accomplish,  using  a 
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reasonably  friendly  software  package.  This  time  issue  is  critical  for 
the  DSS.  The  people  who  will  be  queried  in  this  process  at  Tinker  AFB 
are,  f irst-and-foremost,  flyers  who  have  an  erratic  and  very  full  flying 
schedule.  Attempting  to  corral  experts  to  repeat  the  MADA  approach  for 
each  of  the  12  crew  positions,  using  multiple  factors  and  repeating  the 
process  multiple  times  as  the  system  evolves,  would  probably  kill  the 
DSS  development  outright. 

The  reason  the  process  consumed  so  much  time  was  that  the  experts 
being  consulted,  in  general,  have  no  background  in  probability.  Both 
the  utility  and  weighting  elicitation  process  are  heavily  dependent  on 
the  expert's  ability  to  assign  probabilities  to  the  choices  presented 
during  the  elicitation  process.  The  major  problem  encountered  was  that, 
even  using  a  50-50  gamble  (which  Holloway  indicated  is  generally  the 
most  easily  understood),  the  decision  maker  being  queried  could  not 
build  a  utility  curve  that  matched  his  estimations,  even  Uien  he  had  a 
clear  perception  of  {^at  the  curve  should  look  like.  People  in  general 
do  not  deal  with  probabilities  well,  and  this  experience  indicates  they 
have  particular  problems  estimating  the  relative  probabil it ies  for  two 
different  bounded  ranges  (not  including  either  extreme)  in  the  interior 
of  the  space  they  are  considering. 

For  these  reasons,  MADA  was  removed  from  consideration  for  this 
application  in  the  DSS.  However,  MCDM  in  general,  is  a  tool  that  will 
necessarily  be  appl ieo  to  other  areas  of  the  final  DSS.  This  is 
discussed  in  chapter  5. 
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Multiple  Regression  with  Binary  Responses.  During  the  examination 
of  MCDM  techniques  the  idea  of  using  a  multiple  regression  approach  with 
a  bina'^y  response  surfaced  as  an  appropriate  technique  for  experience 
modeling  .  One  fact  that  was  uncovered  uhile  surveying  experts  for  the 
MADA  evaluation  was  that  a  fairly  rigorous  categor izat ion  based  on 
expert  assessment  of  crew  member  expertise  already  occurred  in  AWACS. 
This  being  so,  there  existed  a  binary  response  that  categorized  crew 
members  as;  (I’s)  —  definite  experts  or  as  (O’s)  —  the  rest  of  the 
crew  force.  (All  experts  do  not  necessarily  fall  in  the  expert 
category).  With  a  binary  response,  a  multiple  regression  is  feasible 
CNeter,  et  al ;  354-3611. 

While  a  multiple  regression  is  feasible  for  the  binary  categorical 
response,  there  are  two  problems.  First,  two  of  the  assumptions  of 
least  squares  regression  do  not  hold.  Least  squares  regression  assumes 
that  the  error  in  the  regression  is  normally  distributed  with  a  mean  of 
zero  and  with  constant  variance.  Since  the  response  is  binary,  the 
error  in  the  regression  can  only  take  on  two  values.  This  being  the 
case,  the  error  is  not  normal.  Similarly,  the  variance  of  the  error  is 
dependent  on  the  response  level,  and  is  therefore  not  constant. 

Second,  there  are  constraints  on  the  possible  values  of  the  response. 

The  interpretation  of  the  output  of  the  regression  model  is  somewhat 
different  than  in  other  regression  formulations.  Since  the  response  is 
binary,  the  expected  value  produced  by  the  regression  model  is  Just  the 
probability  that  a  particular  combination  of  regressors  will  result  in 
an  "expert"  classi ficat ion.  Since  the  output  of  the  response  function 
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is  a  probability,  it  is  constrained  to  the  region  from  0  to  1. 


These  problems  can  be  successfully  dealt  with.  The  lack  of 
constarit  variance  can  be  corrected  for  by  using  a  weighted  least  squares 
fit.  The  constraint  issue  can  be  dealt  with  by  ensuring  that  the 
response  cannot  fall  below  zero  or  above  one  over  the  range  of  the 
predictors  or  by  transforming  the  predictors.  Finally,  while  the  error 
terms  are  not  normal,  for  sufficiently  large  samples  inferences  about 
the  regression  results  can  be  made  as  if  the  error  were  normally 
distributed.  (In  general,  sample  sizes  greater  than  30,  are  sufficient 
for  treating  the  results  as  having  come  from  normal  populations  CDevore 
:  2601.  Samples  for  all  crew  positions  are  larger  than  this). 

The  approach  just  discussed  is  feasible  and  easily  implemented. 

It  is  transparent  to  the  user  in  that  it  does  not  consume  valuable 
expert  time,  but  its  assumptions  can  easily  be  documented  for  expert 
review.  It  was  therefore  chosen  as  the  nodel ing  technique  to  be 
employed  in  the  kernel  DSS.  Implementation  is  discussed  in  the 
remainder  of  this  chapter. 

Kernel  Inplementatian 

The  kernel  was  implemented  for  test  purposes  using  Borland 
International's  Qjattro  Pro  spreadsheet  environment.  This  environment 
was  chosen  for  multiple  reasons.  First,  Qjattro  Pro  acts  directly  on 
multiple  file  formats,  one  of  v^ich  is  DBASE  III.  An  abbreviated  form 
of  the  standardized  personnel  management  database,  which  is  based  on 
DBASE  III  (and  v^iich  was  discussed  in  chapter  3)  was  available  to 
develop  a  prel imi lary  kernel  DSS. 
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Second,  as  compared  to  EJJABLE,  which  was  used  in  the  previous  DSS 
in  this  area,  Qjiattro  Pro  has  much  better  response  times.  Borland's 
memory  management  system  used  in  GLiattro  Pro  is  much  more  sophisticated 
than  ENABLE'S,  significantly  improving  GLiattro  Pro's  speed.  Possibly 
more  important  Quattro  Pro  does  not  support  all  of  the  capabilities  that 
ENABLE  does,  reducing  its  overhead,  and  its  graphics  capabilities  are 
much  better  integrated  to  the  spreadsheet  than  ENABLE' s  were. 

Finally,  Quattro  Pro  is  almost  totally  configurable  to  specific 
applications.  Full  menuing  capabilities  (that  replace  the  normal 
spreadsheet  environment  menus)  are  provided,  multiple  windows  are 
available  for  changing  displays  rapidly,  and  mice  are  fully  supported 
(this  feature  was  not  used).  As  such,  the  user  environment  in  Quattro 
Pro  could  be  tailored  to  match  the  environment  envisioned  in  the 
storyboard  almost  totally  (the  main  exception  was  to  place  the  system 
support  options  as  a  menu  item  on  the  top  menu  bar). 

Pulling  the  Pieces  Together.  In  the  kernel  DSS,  multiple 
regression  with  binary  responses  was  conducted  in  a  PC  based  statistics 
package,  Statistix  II,  using  the  abbreviated  database  mentioned  above. 
(Results  of  this  regression  are  discussed  in  Appendix  E) .  This  modeling 
technique  is  implemented  with  relaxed  tests  for  confidence  intervals,  as 
suggested  by  Deming  CDeming].  (This  issue  is  discussed  in  Appendix  E). 

After  'examining  the  regression  results  and  determining  an 
appropriate  model,  the  appropriate  database  fields  were  read  into 
GLiattro  Pro,  where  the  model  was  applied  to  each  record  of  the  database 
to  build  an  experience  score. 
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Scores  in  the  database  were  separated  into  three  distinct  groups: 
people  who  were  known  to  be  departing  AUIACS,  people  i*4io  were  currently 
not  in  AWACS  (U^ose  records  were  in  the  inactive  portion  of  the 
database)  but  ^«^o  nad  been  gone  long  enough  to  be  due  to  PCS,  and  the 
rest  of  the  people  in  the  crew  position.  The  scores  for  each  group  were 
then  aggregated  using  GLiattro  Pro’s  capability  to  examine  frequency 
distributions,  and  the  results  written  to  a  histogram  to  display  the 
score  distribution.  (This  graph  is  part  of  the  spreadsheet.  A 
transition  to  another  program  module  is  not  necessary  before  it  can  be 
viewed,  in  contrast  to  ENABLE).  The  resulting  graph  was  similar  to  that 
shown  in  Figure  7. 

The  results  of  this  effort  were  a  system  that  operated  and  which 
provided  an  understandable  display  for  the  current  experience  of  the 
crew  position  profiled.  Building  the  system  in  Quattro  Pro  involved  two 
deviations  from  the  design  specified  by  the  storyboards.  First,  the 
user  interface  defined  in  the  storyboards  was  modified.  It  was  built  as 
specified  with  the  exception  of  placing  the  system  support  commands  in  a 
sub-menu  off  of  the  decision  command  menu  and  placing  the  memory  aid 
line  below  the  main  menu,  rather  than  above  it. 

It  was  not  absolutely  necessary  that  the  system  support  commands 
be  moved  to  the  top  menu;  a  menu  at  the  bottom  of  the  screen  could  have 
been  implemented.  However,  flexibility  in  selecting  from  this  menu 
would  have  been  limited,  and  would  have  "felt"  different  from  the 
selection  process  used  in  the  rest  of  the  system  (i.e. ,  mouse  and  cursor 
selection  of  the  system  support  options  on  a  bottom  of  trie  screen  menu 
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would  not  have  been  possible). 

In  contrast,  the  memory  aid  line  had  to  be  moved.  This  is  because 
Qjattro  Pro  reserves  the  line  above  the  main  menu  line  for  its  own  use. 

The  other  deviation  from  the  storyboards  involved  placement  of  the 
two  graphs  being  displayed.  For  resolution  reasons,  the  graphs  were 
placed  one  below  the  other  instead  of  side-by-side.  While  GLiattro  Pro 
is  exceptionally  flexible,  it  does  place  some  restrictions  on  the  user, 
□he  of  them  is  the  lack  of  control  offered  to  the  user  over  some  of  the 
"cosmetics"  the  system’s  graphics.  Large  boarders  are  placed  around 
graphs;  this  required  that  the  two  graphs  be  stacked  if  they  were  to  be 
viewed  at  the  same  time.  Further  comments  on  Quattro  Pro’s  usefulness 
as  a  DSS  environment  can  be  found  in  the  following  chapter,  uhich 
details  the  conclusions  and  recommendations  that  resulted  from  this  DSS 
research  effort. 
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Background 

Uhen  a  request  for  an  ^IT  thesis  proposal  was  received  by  the  28 
AD  in  early  1967,  the  author  proposed  that  the  issue  of  supporting  AUiACS 
personnel  managers  be  examined.  The  proposal  was  prompted  by  the  lack 
of  understanding  of  long  term  personnel  issues  exhibited  by  PFT 
attendees  during  the  course  of  numerous  PFT  conferences.  Mten  the 
proposal  was  acted  upon  by  Schneider,  the  author  was  the  point  of 
contact  at  the  28  AD  for  the  research  effort,  acting  as  the  user's 
"representat i ve" . 

Many  of  the  objectives  outlined  in  the  original  proposal  were 
never  addressed  (prior  to  embarking  on  a  thesis,  the  author  had  no 
appreciation  of  the  constraints  that  limit  a  thesis'  scope).  Further, 
the  majority  of  the  issues  that  were  addressed  were  approached  from  a 
perspective  alien  to  that  envisioned  when  the  proposal  was  made  (i.e. , 
Schneider's  research  addressed  the  problem  largely  from  the  perspectives 
of  TAG,  not  the  28  W). 

The  ways  in  which  the  previous  research  was  limited,  the 
directions  it  took,  and  the  apparent  inability  of  the  "user"  to 
influence  the  course  of  the  research  were  never  understood  by  the  author 
and  prompted  him  to  re-address  the  problem  in  this  thesis.  The  process 
of  viewing  the  same  problem  from  two  perspectives  has  been  an  eye 
opening  experience,  and  has  certainly  impacted  the  types  of  conclusions 
and  recommendations  that  resulted  from  this  research. 

This  chapter  contains  the  conclusions  and  recommendations  that 
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resulted  from  this  research  effort.  The  observations  documented  here 
are  separated  into  two  major  groups:  1)  those  that  deal  with 
development  of  D65s  in  general,  and  2)  those  that  address  the  future  of 
the  specific  DGS  examined  in  this  research. 

G&ieral  DBS  Develqoment 

1)  DSS  development  and  separation  from  the  user^s  environment. 

Conclusions.  DSSs  cannot  be  successfully  developed  at  a  location 
removed  from  the  user.  This  conclusion  results  from  three  areas  of 
difficulty  which  are  discussed  next. 

First,  communications  and  accountability  problems  arise.  M^ile 
storyboarding  is  an  outstanding  communications  tool  (discus&'-<i  later  in 
this  chapter),  it  cannot  correct  the  communications  problems  related  to 
timely  feedback.  Without  constant  communication,  a  necessary  sense  of 
accountability  between  the  developers  of  a  system  (designer,  user  and 
builder)  is  lost,  resulting  in  a  design  process  that  wanders.  (This 
issue  is  also  discussed  later  in  this  chapter). 

Second,  if  the  DSS  is  developed  away  from  the  user's  environment, 
the  system  will  assume  the  designer's  character,  regardless  of  the 
designer's  good  inventions.  Without  constant  communication  and 
immediate  feedback,  the  designer  has  to  act  on  his  perceptions,  with  the 
result  that  the  designer's  views  are  built  into  the  system.  (At  the 
very  least,  the  adaptive  development  cycle  is  prolonged  while  the  user 
weeds  out  those  elements  that  don't  support  his  needs). 

Third,  constant  interaction  with  the  user  is  the  only  way  to 
determine  what  he  knows.  A  general  complaint  of  system  builders  is  that 
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users  do  not  know  vi^at  they  want.  After  the  experiences  involved  in 
this  research,  another  observation  should  be  "users  do  not  know  what 
information  they  have  that  is  applicable  to  the  solution  of  their 
problem". 

All  three  of  these  difficulties  are  at  the  root  of  why  the  author 
didn't  understand  the  course  Schneider's  research  took,  which  resulted 
in  his  pursuit  of  the  same  research  area. 

Further  Observations.  Valusek  makes  a  distinction  between  rapid 
prototyping  and  adaptive  design  CValusekl.  Both  design  methods  are 
geared  to  developing  portions  of  a  system  and  "growing"  the  final  system 
by  merging  the  previously  delivered  portions.  The  distinction  that 
Valusek  makes  is  that  rapid  prototyping  occurs  at  the  developer's 
facilities  and  is  "delivered"  to  the  user  in  pieces,  whereas  adaptive 
design  occurs  at  the  user's  facilities.  This  difference  is  critical, 
and  makes  one  wonder  if  rapid  prototyping  isn't  traditional  systems 
development  in  disguise.  If  the  designers  and  builders  are  not  at  the 
user's  facility,  they  cannot  "wallow"  in  the  problem  or  determine  what 
data  the  user  actually  has  that  may  be  applicable  to  the  problem.  They 
have  to  totally  rely  on  the  user's  description  (read  specification)  of 
his  needs.  Separating  a  DSS's  development  from  the  user's  environment 
results  in  a  rapid  prototyping  approach,  and  entails  building  the  DBS  to 
a  "spec"  in  that  the  system  is  built  to  some  static  standard  between 
deliveries,  just  as  is  done  in  traditional  systems  development.  (Not 
only  is  the  system  being  built  to  "specs",  it  is  being  built  to  a  spec 
multiple  times) . 
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Adaptive  design  is  geared  to  getting  away  from  specs,  or  at  the 
very  least,  letting  the  spec  “float".  The  constant  interaction  between 
the  system  developers  results  in  a  more  "even  evolution"  of  the  system, 
rather  than  a  process  that  involves  taking  two  steps  forward  and  falling 
one  step  back  when  it  is  discovered  "too  much  progress  was  made"  (in  the 
wrong  direction). 

Both  Schneider's  and  this  research  effort  to  support  the  AWACS 
personnel  acquisition  process  are  guilty  of  falling  back  to  a  rapid 
prototyping  approach,  particularly  when  they  are  viewed  together.  This 
research  was  pursued  because  Schneider's  D6S  evolved  in  a  direction  that 
did  not  suit  the  users,  which  required  it  to  be  corrected.  The  DSS 
developed  as  a  result  of  this  research  may  be  equally  guilty.  The  only 
correction  for  this  situation  is  for  any  further  development  of  the  DSS 
to  occur  at  the  2BAD,  where  communication  with  the  user  and  system 
assessment  by  the  user  can  occur  chjring  development. 

Conclusion.  Research  into  DSS  development,  particularly  with 
regard  to  adaptive  design,  needs  to  be  done  at  a  real  user's  site,  and 
removed  from  the  purely  academic  environment.  This  is  a  direct 
corollary  of  the  above  discussion. 

Recomwendations.  This  DSS  should  not  be  examined  for  further 
development  in  thesis  efforts.  All  future  evolution  of  the  system 
should  be  done  by  28  AD  analysts  (28  AD/XOS).  Uhile  the  analysts  at  the 
28  AD  may  not  have  familiarity  with  the  Operations  Research  tools  that 
bear  on  the  problem,  learning  specific  tools  is  easily  corrected.  The 
difficulties  in  acquiring  data  and  understanding  of  user's  needs  from  a 
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distance  is  not. 


2)  S^torvtxMirdinQ  as  a  communications  tool. 

Conclusian.  Storyboarding  is  a  tremendous  tool  for  clarifying 
user  desires  and  designer  intentions.  The  author’s  first  exposure  to 
storyboards  was  during  Schneider’s  research  in  the  AUIACS  acquisitions 
area.  At  all  times,  the  intent  of  the  designer  (Schneider)  was  clear  to 
the  user  (Holt),  and  desires  for  the  system  could  be  clearly  stated  by 
referencing  desirable  and  undesirable  features  of  the  storyboards.  (As 
discussed  above,  however,  the  power  of  storyboards  to  communicate 
intentions  and  desires  was  insufficient  to  overcome  other  communications 
obstacles) . 

Conclusion.  Storyboards  fill  a  very  fundamental  role  in  the 
adaptive  design  process.  Determining  user  requirements  is  a  difficult 
process  that  entails  two  distinct  phases  (IRD  and  IRA).  Both  the  IRD 
and  IRA  processes  can  be  well  structured  by  storyboarding.  The 
storyboards  lend  a  "tactile"  element  to  the  study  information 
requirements.  Users  and  desi^iers  can  "grasp"  their  needs  because  they 
are  clearly  and  simply  stated,  rather  than  floundering  to  an 
understanding  of  needs  while  wading  through  a  morass  of  verbiage  that 
obscures  the  communications  of  needs.  Pictures  ere  worth  a  thousand 
words. 

tecoiwnendation.  An  effective,  general  pxirpose  storyboarding  tool 
should  be  developed.  The  ability  to  build  a  running  "dummy"  system  in 
storyboard  form  that  responds  to  user  inputs  would  be  a  valuable  system 
development  tool.  Further,  it  could  significantly  speed  up  the 
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storyboarding  process,  by  placing  the  storyboards  in  an  environment 
geared  to  building  and  editing  them.  (Word  processors  are  not  effective 
environments  for  building  storyboards). 

3)  DSS  design  as  an  individual  effort. 

Conclusion.  The  importance  of  synergy  can  never  be  overrated. 
Dealing  with  complex  issues  as  individuals  is  not  as  productive  as 
dealing  with  it  as  a  team,  if  the  team  is  built  from  the  right  elements. 
In  the  adaptive  design  context,  the  design  team  can  be  considered  to 
consist  of  the  designer,  builder  and  user.  Together  they  capture  user 
requirements,  determine  user  capabilities,  identify  useful  data,  choose 
applicable  modeling  techniques,  identify  technical  constraints  that  may 
impact  a  system’s  development,  ...  and  address  the  many  other  functions 
that  must  be  considered  in  a  design  process. 

Each  of  these  people  brings  his  own  domain  expertise  to  the  design 
process.  However,  the  user  comes  up  short  in  the  domain  area  when  the 
design  process  leaves  the  "this  is  what  I  want"  stage  and  enters  the 
"how  do  we  give  him  what  he  wants"  stage.  The  user’s  domain  expertise 
is  crucial  to  a  system’s  development.  It  is,  after  all,  his  problem  that 
is  being  addressed.  But  for  systems  design  to  transition  successfully 
to  implementation,  many  questions  need  to  be  answered  in  areas  where  the 
user  has  no  expertise,  and  more,  does  not  even  understand  the  language. 
For  these  questions  to  be  resolved  effectively,  designers  and  builders 
need  to  communicate  with  others  who  speak  the  language  of  systems 
development  so  they  can  sharpen  their  insights  and  ferret  out  their 
errors  and  false  assumptions.  The  lack  of  a  "sounding  board"  to  help 
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sharpen  insights  was  a  significant  handicap  in  approaching  this  research 
as  a  lone  designer. 

tecomnendat ion.  DBS  development  should  proceed  from  a  team 
perspective,  where  the  team  consists  of  multiple  designers,  builders, 
and  users,  not  one  of  each. 

Conclusion.  A  clear  distinction  in  the  functions  of  each  member 
of  a  design  team  is  a  necessity,  and  builders  should  never  be  confused 
with  designers. 

This  research  was  conducted  by  a  "builder",  i.e. ,  someone  used  to 
dealing  with  the  solution  of  problems,  not  their  formulation.  Builders 
assume  that  if  somebody  has  a  problem  to  be  solved,  they  know  what  the 
problem  is  and  how  they  want  it  solved.  Their  Job  is  to  implement  the 
desired  solution.  Therefore,  builders  view  the  system  as  the  product, 
totally  overlooking  or  ignoring  the  fact  that,  for  a  DSS,  the  decision 
is  the  product.  Building  becomes  the  important  task,  resulting  in 
design  functions  being  subsumed  by  building  functions. 

Builders  also  tend  to  look  at  problems  from  a  depth  first  rather 
than  breadth  first  perspective.  They  focus  on  small  portions  of  a 
problem  and  deal  with  it,  approaching  problem  solution  "modularly".  A 
consequence  of  this  approach  is  that,  until  it  is  dealt  with 
successfully,  the  smaller  problem  becomes  "the  problem".  This  cripples 
a  builder's  ability  to  act  as  designers,  since  the  designer  must  be 
concerned  with  the  entire  problem  area. 

To  overcome  these  problems,  there  must  be  a  clear  distinction  as 
to  who  the  designer  is,  and  that  person  cannot  be  the  builder.  Design 
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functions  are  too  important  to  be  entrusted  to  sooiebody  with  a  builder’s 


perspective. 

ftecornnendaticn.  Separate  responsibilities  —  and  never  let  a 
builder  be  responsible  for  design  functions  (otherwise  "design"  will 
occur  in  an  erratic  manner). 


4)  DSS  development  and  technical  constraints. 

Conclusion.  Because  of  the  speed  with  which  basic  blocks  of  a  DSS 
can  be  examined,  spreadsheets  allow  rapid  assessments  of  the  concepts 
embodied  in  the  storyboards. 

The  value  of  the  spreadsheet  environment  lies  in  the  speed  with 
which  a  system  can  be  prototyped.  For  the  DSS  developed  in  this 
research,  a  spreadsheet  (specifically  Quattro  Pro)  was  flexible  enough 
to  implement  the  entire  system's  control  structure  (embedded  in  the 
menuing  system  and  the  memory  aid  system)  as  specified  in  the 
storyboards,  with  only  minor  deviations.  The  environment  also  allowed 
the  disparate  data  necessary  for  the  kernel’s  functions  to  be  viewed  and 
manipulated,  with  the  necessary  models  being  built  into  the  spreadsheet. 

Conclusion.  Spreadsheet  environments  are  currently  inadequate  for 
final  implementation  of  DSSs.  (Xirrent  user  programming  capabilities 
(macros)  in  spreadsheets  are  not  powerful  enough  to  build  systems  that 
need  minimal  support.  This  is  particularly  apparent  when  external 
databases  are  being  queried  by  the  spreadsheet. 

For  this  DSS,  building  a  general  database  interface  with  a  great 
deal  of  conditional  flexibility  was  needed  if  a  system  were  to  be  built 


that  required  minimal  modification  of  the  core  spreadsheet  environment. 


Model  specifications  (that  were  subject  to  change)  needed  to  be  passed 
to  the  spreadsheet.  The  sa/ne  was  true  for  data  specifications  (which 
were  dependent  on  the  model  specified).  Since  both  of  these  elements 
were  subject  to  change  in  dimensionality,  the  specification  of 
calculation  areas  in  the  spreadsheet  for  specific  modeling  operations 
became  a  messy  issue  that  could  only  be  solved  by  execution  of 
conditional  programs  that  exceeded  the  capacities  of  the  macroing 
language  of  the  spreadsheet. 

tecomwmdat ion.  Uhtil  such  time  as  effective  DBS  generators  are 
available,  spreadsheet  environments  should  always  be  considered  for 
quick  assessments  of  DSS  design  elements. 

Further  Gbservat ions.  Development  environments  not  specifically 
designed  to  support  design  in  a  particular  application  area  are 
dangerous  to  the  overall  design  process.  They  impose  technical 
constraints  on  a  system's  development  that  may  not  be  appropriate. 

Worse,  they  breed  commitment  to  the  system  in  its  current  stage  of 
development,  when  this  commitment  is  not  Justified.  It  is  v'ery  hard  for 
a  system’s  developer  to  abandon  past  development  because  of  the 
constraints  of  the  environment  he  is  operating  in;  it  is  easier  to 
change  the  "requirements"  of  the  system  to  be  developed.  This  is 
particularly  apparent  in  the  DSS  Schneider  designed  for  AW^CS. 
Schneider’s  environment,  ENABLE,  imposed  significant  constraints  on  the 
DSS  design  (this  is  conjecture  on  the  authors  part,  but  is  consistent 
with  his  experience,  and  goes  a  long  way  to  explaining  the  course 
Schneider’s  DSS  took). 
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Conclusicn.  DSS  development  in  the  adaptive  design  paradigm 
without  effective  DSS  generators  is  severely  crippled.  Turn  around  time 
for  useable  system  enhancements  are  simply  not  short  enough  for  the 
adaptive  design  process. 

Uhtil  such  time  as  a  development  environment  is  available  that  is 
flexible  and  extensive,  development  will  have  to  occur  in  one  of  two 
environments.  First,  design  can  occur  in  spreadsheets,  as  discussed 
above,  where  assessments  of  DSS  design  elements  can  rapidly  be  made. 

Ulien  the  design  element  has  been  "shaken  down",  it  can  then  be 
implemented  in  stand  alone  code  for  integration  into  a  final  system. 

Uh fortunately,  this  is  very  w^asteful  of  the  effort  that  goes  into 
developing  the  spreadsheet  capability. 

The  other  possibility  is  to  build  the  system  in  stand  alone  code 
from  the  start.  This  is  a  possibility,  but  only  if  7 arge  programming 
toolboxes  applicable  to  the  task  are  available.  Without  the  necessary 
toolbox  (which  constitutes  the  beginnings  of  a  DSS  generator), 
development  time  for  the  basic  system  components  will  make  delivery  of 
DSS  components  impossible  in  adaptive  design  time  frames. 

Rfecumnendat ion.  A  major  research  effort  should  be  started  in  the 
area  of  developing  specifications  for  DSS  generators. 

5)  The  critical  nature  of  assumptions. 

Conclusion.  The  critical  nature  of  assumptions  cannot  be 
overstated  for  DSSs.  When  a  user  knows  what  decision  needs  to  be  made 
and  resorts  to  a  DSS  to  arrive  at  his  decision,  he  has  lost  control  of 
the  decision  process  and  rendered  the  decisions  arrived  at  suspect  if  he 
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does  not  know  v>iat  assumptions  have  been  implemented  in  the  DBS’s 
design.  All  modeling  techniques  entail  assumptions;  some  are  related  to 
how  data  is  treated,  and  some  are  related  to  the  applicability  of  the 
various  modeling  techniques  used.  All  of  the  assumptions  made  using  the 
DBS  modeling  technique  need  to  be  “examinable"  if  the  decision  maker  is 
to  make  valid  decisions. 

F^ommendation.  Some  structure  similar  to  the  "ASSUMPTIONS" 
button  presented  by  El  Sherif,  et  al,  needs  to  be  incorporated  in  all 
DSSs,  and  all  assumptions  need  to  be  fully  documented. 

The  aiecific  DBS 

6)  This  DBS  and  vhere  to  go  in  the  future. 

Conclusion.  Expt;'rt  opinion  is  going  to  play  a  major  part  in 
v>iatever  quantification  scheme  is  proposed  for  the  AWACS  capabilities 
assessments.  This  is  also  true  for  determining  an  appropriate  basel ine 
for  desired  experience  distributions  in  the  different  crew  positions. 
This  is  because  there  are  currently  no  hard  and  fast  measures  for  these 
areas.  MCDM  techniques  are  very  relevant  to  this  area. 

Conclusion.  Synergy,  as  discussed  for  design  team  issues,  also 
exists  among  modeling  techniques.  The  examination  of  MADA  for 
quantifying  experience  led  to  identification  of  criteria  by  which  a 
subset  of  a  crew  pos.tion  could  be  categorized  in  experience.  The 
regression  technique  specified  for  use  in  the  kernel  (which  overcomes 
the  time  issues  that  make  application  of  MCDM  technique's  to  the  problem 
untenable)  was  only  able  to  be  used  because  of  information  uncovered 
using  MADA  techniques. 
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Reconmendaticn.  Multiple  nodel ing  techniques  should  be  examined 
for  appl icabil ity  to  each  phase  of  this  DSS's  development. 

Conclusion.  The  regression  method  employed  in  the  kernel  has  some 
potential  stumbling  blocks. 

As  discussed  in  the  previous  chapter,  regression  results  only  make 
sense  if  the  response  is  bounded  by  zero  and  one.  When  a  set  of  factors 
is  determined  to  be  significant  in  predicting  membership  in  the  "expert 
class"  (defined  by  StanEval  personnel  and  instructors),  the  range  of  the 
factors  needs  to  be  examined  to  ensure  that  they  don’t  force  the 
response  into  an  un-al lowed  range.  If  infeasible  responses  are 
encountered,  transformations  of  the  predictors  may  be  necessary.  Another 
possible  strategy  is  to  examine  the  predictors,  to  see  if  it  makes  sense 
to  "cap”  the  values  they  can  take  an.  For  example,  if  flying  hours  are 
a  factor  in  the  regression  model,  it  may  be  that  it  makes  no  sense  to 
count  flying  hours  above  2000  hours.  Marginal  flying  hours  above  this 
number  may  not  impact  experience  significantly.  This  type  of  "call"  is 
an  expert  judgement  call  —  MCDM  techniques  may  be  useful  in  evaluating 
these  types  of  questions. 

(^commendation.  Always  keep  in  mind  that  regression  (like  all 
other  modeling  techniques)  only  provides  guidance.  The  analyst  must 
bring  his/her  judgement  to  bear  on  what  the  results  mean,  and  must  also 
keep  data  implications  in  mind,  if  effective  models  are  to  be  built. 

Conclusion.  This  DSS  is  going  to  require  extensive  support  from 
the  28  AD  analysis  office.  Implementing  the  entire  system  discussed 
here  will  be  a  major  undertaking.  This  effort  is  one  that  the  28  AD 
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analysis  shop,  as  it  is  currently  configured,  is  uell  equipped  to 
address,  given  the  combination  of  analysts,  programmers  and  a  supervisor 
with  extensive  systems  development  background.  However,  the  commitment 
required  from  this  office  is  going  to  extend  well  beyond  "system 
delivery",  where  system  delivery  refers  to  a  time  uhen  it  appears  that 
the  system  has  evolved  to  a  fairly  stable  state.  At  this  point,  \»hen  it 
appears  that  user  needs  are  being  met,  support  requirements  for  this 
system  are  still  going  to  he  extensive. 

As  changes  in  the  crew  force  come  about  due  to  changed  acquisition 
practices,  it  must  be  realized  that  models  developed  to  address  earlier 
crew  compositions  will  have  to  change  —  and  regression  camct  be 
automated.  As  discussed  above,  the  analyst  is  an  integral  part  of  the 
regression  modeling  approach,  and  will  therefore  have  to  be  an  integral 
part  of  the  DSS’s  operation.  The  analyst  must  be  involved  to  interpret 
the  regression  results  and  to  determine  the  most  effective  model  to 
describe  v^at  is  observed. 

Sunwarv 

As  a  learning  experince,  the  value  of  this  research  cannot  be 
overstated;  its  impact  on  the  author  has  been  immense.  The  different 
natures  of  design  and  building  were  "discovered"  (and  constantly  re¬ 
discovered)  during  the  research,  resulting  in  the  realization  that  good 
design  does  not  often  occur  by  happenstance,  it  is  hard  work. 

For  the  (Operations  Researh  world,  the  insights  pertaining  to 
division  of  labor  in  DSS  development  and  the  necessity  for  good  DSS 
development  tools  are  important,  and  are  areas  requiring  further  study. 
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Appendix  A 


Concept  Map 

The  concept  map  describing  the  AWACS  personnel  acquisition  process 
is  included  on  the  following  two  pages.  This  concept  map  is  presented 
in  a  hand  drawn  form  for  two  reasons.  First,  a  concept  map  is  only  a 
guide  to  understanding  the  intricacies  of  the  problem  being  addressed  — 
it  should  never  be  vieu/ed  as  a  product  that  defines  the  problem. 

A  "perfect"  concept  map  could  be  construed  to  be  a  product  ubiich 
fully  describes  a  problem  and  implies  that  the  problem  is  completely 
understood.  This  is  nonsense,  and  could  stifle  further  exploration  of 
the  problem.  Making  the  concept  map  "perfect"  is,  therefore,  counter 
productive.  (Also,  effort  expended  making  the  concept  map  "pretty"  can 
better  be  spent  addressing  how  to  solve  the  problem  which  the  concept 
map  describes. 

Above  all,  the  concept  map  must  be  viewed  for  what  it  is  —  a 
useful  tool,  not  a  product  in  its  own  ri^t. 
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Figure  11  (part  b)  -  Concept  Map  for  AUIACS  Personnel  Acquisitions 
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^^ppendix  B 


StorytxDards 

The  storyboards  describing  the  AWACS  PERSOM'JEL  MANAGEMENT 
ACQUISITION  SYS131  follow.  They  are  divided  into  five  functional  areas 
that  indicate  the  issues  they  address  in  the  system’s  operation.  These 
areas  are; 

1)  System  functions  —  addressing  how  the  system  is  structured 
and  how  it  is  used; 

2)  Assessments  of  current  crew  position  profiles  —  the  user 
needs  to  understand  ^«bere  he  currently  stands  before  he  attempts  to  plan 
acquisitions; 

3)  Acquisition  planning  and  assessment  of  long  term  impact  —  the 
user  schedules  acquisitions  under  different  assumptions  and  examines  the 
impact  the  planned  acquisition  has  on  the  crew  position; 

4)  Assessment  of  personnel  issues  on  mission  capabilities  —  the 
manager  needs  to  know  how  to  balance  conflicting  acquisition  needs. 
System’s  performance  provides  the  measure,  and; 

5)  Provisions  for  the  managers  to  examine  low  level  information 
that  impacts  their  decision  —  e-g.»  examining  the  personnel  databases 
and  accessing  information  on  scheduled  changes  to  AWACS  manning  that 
will  impact  his  needs  assessments. 

Each  storyboard  has  a  description  of  its  major  features.  These 
descriptions  are  not  stated  explicitly  in  terms  of  ROMC.  Rather,  the 
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descriptions  are  in  terms  of  what  the  user  needs  to  know  to  determine 
that  the  system  meets  his  needs.  (The  correlation  of  the  description 
elements  to  the  elements  of  RQMC  should  be  self-evident  to  readers 
familiar  with  the  RQMC  structure). 


00 


System  Control  and  Help 
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AUACS  PERSONNEL  ACQUISITION  HANASENENT  SYSTEN 

POSITION 

SCHEDULE 

NANNING  PROFILE 

AUACS  CAPABILITIES 

FORCE  STRUCTURE 

UelcoK  to  the  28  Air  Division  AUACS  Personnel 
Acquisition  Hanageeent  Systee 


Systee  files  are  nou  being  accessed 

Please  wait 

(Approxieate  tiae  :  1  linute) 


Help 

FI 


Notepad 

F2 


Hookbook 

F3 


Assui^tions 


Prior  Screen 
F5 


Main  Menu 
F6 


Print 

F7 


Quit 

FIO 


--  (1) 
-  (2) 


--  (3) 


--  (4) 


12  '  Systee  Structure  Representation 


This  is  the  entry  screen  for  the  support  system.  The  system  is 
specified  on  the  memory  aid  bar  (1)  and  the  major  system  options  are 
designated  on  the  system  command  menu  bar  (2).  The  system  work  and 
display  area  (3)  contains  a  message  welcoming  the  user  informing  him 
that  there  will  be  a  brief  wait  winile  data  files  are  accessed.  The 
major  system  support  functions  are  contained  on  the  bottom  menu,  the 
system  support  menu  bar  (4). 


System  Comtnand  Bar;  The  major  decision  areas  are  separated  from  the 
rest  of  the  system  and  placed  on  the  top  command  bar.  They  are  also  in 
all  caps  as  a  further  reminder  that  they  are  decision  area  commands. 
Descriptions  of  the  commands  can  be  accessed  through  the  help  command 
<F1).  Added  information  for  selected  options  follows. 

POSITION:  The  POSITION  option  allows  the  user  to  filter  the  database 
so  only  the  crew  position  of  interest  is  evaluated  during  the  current 
session.  A  specific  position  must  be  designated  for  the  scheduling 
routine  and  some  of  the  manning  profile  routines.  If  a  position  hasn't 
been  named  prior  to  selecting  the  SCHEDLLE  option  or  the  affected 
subsets  of  MANNING  PROFILE  option,  the  system  will  self-select  the 
POSITION  option  before  continuing. 

SOEDULE:  This  is  where  the  bulk  of  the  work  with  this  system  will 
be  conducted.  Current  position  in  manning  and  future  projections  are 
used  in  a  "What-if"  cycle  to  arrive  at  manning  acquisition  decisions.  A 
significant  feature  of  this  capability  is  the  ability  to  show  TAC/MPC 
the  immediate  and  long  term  consequences  of  their  acquisition  policies, 
in  a  quantitative  format. 

MAMvJING  PROFILES:  This  options  has  a  two-fold  function:  to  provide 
background  at  a  low  level  (a  subset  of  the  actual  database  driving  the 
models),  and  to  allow  the  user  to  make  high  resolution  assessments  of 
the  experience  (current  and  near  term  change)  of  the  crew  position  he  is 
responsible  for. 
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System  ?=>»TnQrt  Bar';  The  system  support  bar  provides  the  support 
required  of  the  system  that  is  not  directly  related  to  the  decision 
elements  of  the  process  being  supported.  These  commands  are  separated 
from  decision  commands  by  position  and  by  the  typographic  form  of  the 
commands. 

Help;  Provides  context  sensitive  insights  and  direction  that  clarify 
the  decision  process  and  the  software's  operation. 

Notepad:  Provides  user  with  a  pad  to  document  his  insights, 
questions,  and  ideas  about  personnel  acquisition. 

Hookbook:  A  notebook  for  system  related  ideas  or  comments. 

(Any  sequence  of  menu  selections  in  a  particular  session  are  stored  in  a 
buffer  and  recorded  as  memory  aid  entries  on  the  notepad  and  hookbook 
whenever  they  are  opened). 

Assumptions;  At  any  point,  the  user  can  query  the  system  to  see 
what  assumptions  have  been  made  in  the  way  data  is  treated.  This  is 
particularly  useful  for  ensuring  the  user  understands  ui^at  models  are 
doing,  and  what  the  weaknesses  of  their  operations  may  be. 

Prior  Screen;  Allows  user  to  backup  1  screen  per  keystroke.  A 
screen  is  defined  as  any  display  that  is  uniquely  different  from  the 
prev'ious  one,  i.e. ,  if  a  display  has  a  sequence  of  commands  performed  on 
it  that  adds  new  material,  each  stroke  of  F5  will  back  the  user  out  from 
the  results  of  the  last  command. 

Main  menu:  Takes  user  to  the  system  command  bar  for  his  next  action. 

Print;  Make  hard  copy  of  current  screen  and  the  materials  being 
referenced,  e.g.,  entire  roster  being  referenced,  of  which  only  a 
portion  is  visible  on  the  screen. 

Ghait:  Returns  user  to  operating  system  or  shell,  with  an  option  to 
save  any  work. 


Major  System  Control  Features:  The  major  control  features  of  the  system 
are  embodied  in  the  menuing  structure  displayed  on  the  screen.  Further 
structure  is  provided  by  limiting  access  to  certain  system  command 
options  v>*ien  others  are  active.  In  these  instances,  commands  that 
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cannot  be  selected  from  within  current  operations  disappear  from  the 


command  menu.  For  example,  when  MAMVJING  PRCFILE  is  selected,  the  only 
other  system  command  option  that  remains  on  the  menu  is  the  POSITION 
option,  allowing  profiles  to  be  specified  by  crew  position  (see  the 
following  story  board). 

Memory  Aids:  Memory  aids  are  also  largely  embodied  in  the  systems 
presentation  format.  Main  system  commands  that  are  currently  active 
remain  highlighted;  sub-level  options  and  the  choices  made  at  the  sub- 
level  will  be  displayed  on  the  Memory  Aids  Bar  (2).  (See  the  third 
story  board  for  clarification).  As  a  further  aid  to  memory,  levels  of 
sub-menus  are  kept  to  a  minimum;  the  user's  understanding  of  where  he  is 
in  the  decision  cycle  in  enhanced  by  keeping  command  trees  short. 
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1  AUACS  PERSONNEL  AC6UIS1TI0N  NANAGENENT  SYSTEH  | 

POSITION 

SCHEDULE 

NANNING  PROFILE 

AUACS  CAPABILITIES 

FORCE  STRUCTURE 

Main  lenu  options  aay  be  chosen  froa  the  systea  aenu  bar  by  selectinp 
the  highlighted  character,  by  pointing  and  clicking  with  a  aouse  or  oy 
placing  the  cursor  in  the  desired  box  and  striking  return. 


POSITION  '  Select  the  crew  position  of  interest.  Default 

is  all  positions  being  evaluated. 


SCHEDULE  -  Plan  personnel  acquisitions  strategies. 


NANNING  PROFILES  -  View  current  aanning,  experience  profiles,  gain 
and  loss  estiaates  and  their  iapacts. 


AUACS  CAPABILITIES  *  Assessaents  of  ANACS  systeas  effectiveness  as 
iapacted  by  personnel  issues. 


FORCE  STRUCTURE  -  View  weapons  systea  factors  driving  personnel 
requireaents. 


Systea  support  bar  options  aay  be  selected  by  designated  F  keys,  and  by 
the  aethods  for  the  aain  aenu  (except  there  are  no  highlighted  letters) 


Help 

Hookbook 

Assuaptions 

Prior  Screen 

Ha in  Henu 

Print 

Quit 

FI 

F3 

M 

F5 

F6 

F7 

FIO 

Figure  13  -  Exaaple  Help  Screen  Decision  Coaaand  Descriptions 
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This  is  the  main  help  screen  that  gives  a  basic  description  of  the 
main  commands  on  the  decision  command  bar.  This  is  the  help  screen  that 
is  activated  if  no  sub-menu  options  have  been  user  selected.  More 
detailed  information  is  available  by  selecting  (via  mouse  or  cursors) 
the  command  of  interest.  In  general,  the  help  system  can  be  accessed  at 
any  time  and  is  context  sensitive,  with  help  being  provided  for  the 
currently  active  command. 
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This  is  the  help  screen  that  is  displayed  if  help  is  activated  vhile  HANNINfi  PROFILE  is 
selected  on  the  decision  conand  bar.  (he  aeeory  aid  bar  indicates  that  the  NANNING  PROFILE  option  is 
indeed  active).  Each  of  the  sub-eenu  options  under  NANNING  PROFILE  is  briefly  described.  Hore 
detailed  descriptions  are  available  by  selecting  the  sub-ienu  option  of  interest  (by  highlighting  the 
appropriate  title). 
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Assessaent  of  Current  Crew  Position  Status 
(Entry  Level  Assessnents) 
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This  is  the  main  entry  screen,  and  is  entered  by  system  default  after 
all  necessary  data  files  have  been  loaded.  The  first  time  into  the 
system,  the  current  aggregated  experience  profile  for  all  AWACS  crew 
positions  is  presented.  After  a  particular  user  selects  the  crew 
position  he  will  use,  the  system  will  subsequently  come  up  with  his 
position  profiled.  (For  high  level  managers,  the  profile  for  all 
positions  is  available  by  setting  the  position  filter  to  "ALL"). 

This  entry  point  to  the  decision  process  was  selected  allow  managers  to 
understand  the  current  experience  profile  of  the  positions  they  manage 
before  becoming  involved  in  acquiring  bodies.  Managers  will  therefore 
know  when  acquisition  planning  starts  ti^iether  special  attention  must  be 
given  to  the  types  of  acquisitions  offered  by  MPC. 

The  manager  gets  two  profiles.  The  left  profile  shows  the  current 
status  of  the  crew  position  and  the  impact  knoun  losses  will  have  on  it. 
The  right  profile  indicates  the  best  profile  available  if  all  known 
"eligible"  former  AUACers  from  this  position  are  recaptured  from  the  Air 
Force  manning  pool.  (If  the  corrections  are  not  enough,  the  manager  may 
have  to  examine  the  possibility  of  c^turing  people  with  applicable 
experience) . 

Position  is  the  only  system  command  menu  directly  available  to  the  user 
from  the  home  display.  Position  filters  can  be  changed  directly  by 
hitting  "P"  or  selecting  "POSITION”  with  a  mouse  or  cursors. 
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Leaving  this  display  requires  F6  to  be  selected  to  enable  access  to  the 
whole  system  command  menu  bar. 
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Selection  of  POSITION  drops  a  pop  down  menu  containing  the  AWPCS 
abbreviations  for  all  crew  positions  on  the  aircraft.  Selection  is  by 
command  letter  (highlighted)  or  by  point  and  click.  The  default 
selection  is  "All". 

Selecting  Help  at  this  point  would  spell  out  the  positions  associated 
with  each  abbreviation,  describe  the  filtering  that  occurs  if  a  position 
is  selected,  and  describe  why  filtering  might  be  desired. 

After  a  position  filter  is  selected,  the  choice  made  will  appear  on  the 
memory  aide  bare  on  all  subsequent  screens  as  long  as  the  filter  is 
active. 
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The  user  operation  executed  in  the  previous  storyboard  selected  MXs  for 
profiling  —  this  is  indicated  on  the  memory  aid  bar.  The  displayed 
profiles  now  reflect  MCX;  experience. 

This  screen  also  provide  baseline  information  that  is  useful  to  the 
manager  v.hen  evaluating  a  crew  position.  Information  on  current 
authorizations  and  manning  are  included  above  the  profiles.  In  this 
instance,  the  MX  manager  can  see  that  he  is  losing  key  experienced 
people,  and  that  he  is  already  short  on  people.  In  the  normal  course  of 
the  acquisition  process,  the  manager  could  expect  to  get  novices  that 
would  enter  the  experience  profile  at  the  low  end  of  the  scale.  I f  he 
corrects  his  initial  manning  deficiency  and  his  expected  losses  with 
this  type  of  acquisition,  he  easily  see  that  the  inexperienced  end  of 
the  experience  profile  will  become  very  heavy.  He  must  do  something  to 
acquire  experienced  people  if  he  is  to  avoid  this  situation. 
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This  is  the  experience  profiling  screen  for  evaluating  the  long  term 
impact  of  the  acquisitions  that  the  manager  plans  for  the  five  year  PFT 
cycle. 

The  experience  level  projections  are  driven  from  the  scheduling  entries 
that  the  manager  makes  during  his  “uihat-if"  examination  of  acquisition 
scheduling  options  (see  the  next  storyboard).  His  control  of  numbers  of 
people  acquired,  their  experience,  Vshen  they  are  acquired,  and  his 
suggested  control  of  factors  that  impact  how  a  position  loses  people  all 
impact  the  projection. 

In  this  instance,  the  manager  was  looking  at  MCCs.  The  left  most  bar  in 
each  experience  group  is  for  the  current  year  (1989  in  this  instance). 
Looking  across  the  different  groups  at  the  left  most  bar  indicates  that 
the  MCC  position  is  heavy  on  inexperience  people.  The  right  most  bar 
gives  the  profile  after  five  years  if  the  managers  acquisition  plan  is 
followed.  In  this  instance,  experience  for  the  position  is  distributed 
more  evenly  across  the  groups,  the  number  of  people  in  the  three  most 
experienced  groups  has  doubled,  and  the  average  experience  of  the  crew 
position  has  increased. 
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Scheduling  Acquisitions 

(and  Nid-level  Assessnents  of  Crev  Position  Status) 
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Selecting  SOEDULE  runs  a  model  that  plots  projected  manning  based  on 
current  manning  and  acquisition  policy  against  the  planned  requirements 
derived  from  FORCE  STRUCTURE  considerations.  The  requirements  are  in 
black  and  the  current  projections  are  in  red.  Levels  for  all  current 
policy  parameters  are  shown  in  the  parameter  box  labeled  "RED  (NOW)". 

A  table  of  labels  for  the  parameters  that  quantify  policy  positions  is 
provided,  along  with  a  corresponding  table  of  current  values.  The 
parameters  are  broken  into  two  categories,  internal  and  external.  The 
internal  policies  are  things  that  the  28  Air  Division  has  control  over, 
and  the  external  policies  are  those  whose  change  would  require 
concurrence  from  outside  agencies,  such  as  TAC  and  MFC.  As  exanples, 
the  28  Air  Division  has  some  control  over  how  it  allocates  flying  hour 
budgets  and  has  some  flexibility  in  how  many  people  it  can  train  in  each 
crew  position  each  year.  However,  controls  the  length  of  the  CXIDE 
55  incurred  for  training,  and  also  determines  the  average  tour  length 
through  its  PCS  actions.  The  breakout  is  provided  so  the  user  knows 
which  parameters  he  has  the  most  control  over. 

The  parameters  that  appiear  in  the  boxes  are  chosen  to  fit  at  least  one 
of  three  criteria.  First, they  can  be  things  that  inpact  acquisition 
capabilities  in  some  way  (such  as  training  capacity  that  limits  how  many 
people  can  be  bought  in  a  year).  Second,  they  can  be  things  that  will 
alter  demand  for  personnel  (such  as  Code  55s,  which  tend  to  impact  the 
average  time  in  AWACS).  Third,  they  can  be  things  that  impact 
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experience  of  the  crew  force  (such  as  number  of  acquired  bodies  in  a 
given  year  with  some  specified  experience  score).  (Actually,  it  is 
necessary  that  some  of  the  parameters  be  directly  related  to  the  factors 
that  the  experience  scoring  model  uses  to  produce  scores.  There  must  be 
some  mechanism  between  the  scheduling  process  and  the  experience 
projections  that  allows  individual  experience  to  be  incremented  as  time 
goes  by). 

Multiple  "v»*iat-if'‘  tables  are  provided  to  allow  the  user  to  make 
comparisons  among  various  settings  of  the  parameters.  For  the  current 
screen,  assume  that  the  current  training  structure  is  set  up  to  train  20 
CDMTs  a  year  and  that  they  incur  a  4  year  CX)DE  55  after  training.  The 
user  can  look  at  the  impact  of  changing  these  values  to  22  and  5 
respectively,  and  observe  the  imp^Krt  on  the  manning  chart.  (The 
parameters  can  be  scrolled  and  are  synchronized  across  the  boxes  to 
facilitate  comparison  of  the  parameter  values  for  a  given  plot).  The 
objective  is  for  the  user  to  come  up  with  an  achievable  set  of 
parameters  that  cause  the  projection  line  to  match  the  requirement  line 
as  nearly  as  possible. 

Each  of  the  "vhat-if"  boxes  is  plotted  in  its  own  color;  a  new  box  will 
be  entered  when  the  SAS/E  command  of  the  current  box  is  selected. 
Previously  saved  boxes  can  be  re-entered  by  backing  up  through  the  boxes 
using  the  Prior  Screen  (F5)  command. 
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Uliile  this  screen's  operations  is  to  schedule  acquisitions,  total 
manning  is  not  the  whole  picture.  Therefore,  the  MfilMMIlMG  Pf3CFILE  and 
AUIACS  CAPABILITIES  menu  options  are  available  to  examine  the  impact  of 
changing  parameters.  Selecting  the  AWACS  CAPABILITIES  option  will  drop 
a  window  over  the  graph  produced  by  SCHEDULE  that  contains  the  same 
capabilities  graph  previously  described,  with  plots  for  the  CURRENT  box 
and  any  ”(<iiat-if"  boxes  with  changed  parameters  in  them.  Seie.tion  of 
the  MAIvNING  PROFILE  option  from  within  SCHEDULE  will  drop  a  window  over 
the  current  plots  and  contain  a  line  graph  of  the  SCHEDULE  and  ANACS 
CAPABILITIES  format.  The  graph  will  indicate  a  reference  "average 
desired  experience  level"  line  for  the  crew  position,  and  display  the 
plots  for  the  CURRENT  box  parameters  and  any  "what-if"  box  parameters 
that  have  been  saved. 

By  assessing  his  choices  graphically  with  direct  comparison  of  the 
results  of  his  choices,  the  user  can  arrive  at  a  "good"  answer,  and 
support  it.  He  knows  what  he  has  considered,  and  has  built  in 
documentation  showing  the  results  of  his  various  choices.  He  also  has 
quantitative  estimations  of  the  consequences  of  any  parameter  value 
forced  on  him  from  outside  the  Division,  e.g.,  TAC  decides  to  buy  only  4 
new  CDMTs  for  a  given  year. 
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Systems  C^seraticns  Impacts 
(End-game  Assessments  of  Acquisition  Strategies) 
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Selection  of  AUIACS  CWP^ILITIES  produces  a  graphic  plot  of  AWACS  mission 
capability  as  affected  by  personnel  issues.  The  plot  is  from  the 
current  date  minus  2  year  to  current  date  plus  5  years  (1  long  range 
acquisition  cycle).  The  capability  factor  is  plotted  relative  to  tuo 
reference  levels  (scaled  to  100):  desired  day  to  day  minimum  capability, 
and  minimum  war  fighting  capability  for  effective  operation.  The  plot 
is  driven  by  a  Multi-O-iteria  Optimization  model  that  accesses  the 
personnel  database.  Major  inputs  are  crew  experience  profiles  and 
absolute  manning  levels. 

The  impact  of  personnel  issues  on  AWACS  capabilities  are  of  paramount 
importance.  This  combined  assessment  of  manning  levels  and  experience 
provides  an  important  tool  to  structure  new  accessions.  Quantity  may 
have  to  be  balanced  by  quality;  re-acquiring  of  previous  AWACS  p)ersonnel 
may  be  the  only  solution  to  a  decrease  in  capability  caused  by  a  lack  of 
experience  or  a  shortage  of  bodies.  The  timing  of  acquiring  new  bodies 
or  attempting  to  re-acquire  old  ones  will  be  critical  in  balancing 
capability  shortfalls  that  occur  in  the  future  due  to  current  personnel 
managements.  The  AWACS  CAPABILITIES  graphics  provides  a  means  for 
identifying  the  shortfalls  and  the  specific  causes. 

The  memory  aid  bar  indicates  there  is  no  crew  position  filter  set.  If  a 
filter  has  been  previously  specified,  selection  of  AWACS  CAPABTLITIEB 


disables  the  filter. 


Systems  capabilities  is  a  composite  estimation  of  ability  to  perform  a 


mission.  No  other  menu  options  are  available  from  this  command. 

A  zoom  feature  is  supported  to  allow  for  increased  resolution  in  a 
selected  area.  This  facilitates  examination  of  critical  areas  in  the 
future.  Selecting  the  Zoom  command  will  position  cross-hair  on  the 
screen  that  can  be  positioned  with  mouse  or  cursors.  A  special  key  to 
execute  Zoom  is  provided;  this  is  as  a  courtesy  to  facilitate  execution 
of  a  command  from  the  live  window.  It  is  also  a  reminder  that  there  is 
more  to  this  screen  than  the  presented  graph.  All  commands  offered  as 
sinale  commands  on  the  live  screen  can  be  executed  by  clicking  or 
striking  the  return  key. 
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Selection  of  the  Zoom  feature  placed  cross-hairs  on  the  screen.  After 
positioning,  clicking,  hitting  return,  or  hitting  F8  executes  the  zoom 
at  the  specified  point. 

F8  option  is  again  the  "hot  key"  from  the  live  screen.  Again,  all 
commands  offered  as  single  commands  on  the  live  screen  can  be  executed 
by  clicking  or  striking  the  return  key. 
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The  screen  displayed  as  a  result  of  the  Zoom  sequence  show  a  roughly  7 
to  1  increase  in  date  resolution  on  a  re-scaled  graph.  The  display 
centers  the  selected  praint  from  the  low  resolution  graph  on  the  new 
plot,  indicated  by  vertical  tics  crossing  the  reference  lines  at  the  0 
coordinate.  The  plot  is  shown  for  six  months  on  each  side  of  the 
selected  point. 

The  fact  that  the  user  is  viewing  capabilities  at  increased  resolution 
IS  provided  as  a  reminder  on  the  memory  aid  bar. 

Discrete  values  for  each  month  are  provided  in  a  table  to  remove 
interpolation  problems.  Integer  parts  are  place  on  one  line  and 
fractional  parts  on  another.  This  improves  readability  of  the  screen, 
and  also  speeds  up  scanning  for  trends,  ignoring  fine  detail. 

Finally  a  table  is  provided,  showing  the  four  crew  positions  that  the 
model  driving  the  assessment  being  plotted  is  most  sensitive  to  at  the 
current  point.  (Sensitive  to  with  respect  to  improvement).  The  crew 
positions  are  rank  ordered. 

Detailed  assessment  of  the  impact  of  specific  crew  positions  is 
available  at  this  point  by  selecting  the  position  of  interest  from  the 
"LIMITINS  POSITIONS"  list. 
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Lcm  level  SMppoirfcing  Information 
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The  FORCE  STRUCTURE  option  is  selected  to  view  the  prime  factors  that 
drive  changes  in  AMACS  manning.  Theses  are; 

PAA:  Primary  Assigned  Aircraft  —  The  number  of  aircraft  used  in 
conducting  AWACS  mission.  Purchases  of  new  aircraft  would  require  more 
manning. 

CREUI5:  Number  of  crews  assigned  for  each  PAA.  Changes  in  AWACS 
mission  could  warrant  changes  in  the  number  of  crews  assigned  to  each 
aircraft. 

AIRCRAFT  MDIFICATIONS:  Changes  in  AWACS  mission  can  require  changes 
in  the  crew  structure.  As  an  example,  the  block  30-35  mod  on  the  E-3 
will  call  for  more  WDs  and  ASTs  to  enhance  weapons  control  and 
surveillance  capabilities. 

ORGANIZATIONAL  CHANGES;  Structural  changes  in  the  28  Air  Division 
may  impact  manning.  The  creation  of  the  962  AWAC  Squadron  increased  the 
overhead  for  certain  crew  positions  to  provide  the  squadron  staff. 

The  memory  aid  bar  indicates  there  is  no  crew  position  filter  set.  The 
POSITION  command  is  present,  indicating  FORCE  STRUCTURE  can  be  filtered 
if  desired.  (If  selected,  the  position  pop  down  menu  appears).  If  a 
position  is  selected,  the  PAA  and  CREW  elements  remain  unchanged.  Only 
the  crew  position  of  interest  would  appear  under  AIRCRAFT  MODIFICATIONS 
and  0R»WIZATI0NAL  CHANGES. 

The  AIRCRWn-  MODIFICATIONS  and  0R(^IZATI0NAL  CHANGES  sub-charts  have 
scroll  bars  to  scroll  the  crew  position  charts,  if  necessary.  Point  and 
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click  (mcxjtse  or  cursor/return)  on  the  up  or  doi*n  arrows  to  scroll  in  the 
indicated  direction,  with  the  current  position  in  the  list  being 
indicated  by  the  left  pointing  arrow.  (As  in  Macintosh  operations). 

The  sub-chart  heading  will  remain  to  indicate  which  sub-chart  is  which. 

The  BASE  column  holds  the  appropriate  values  for  the  current  force 
structure.  In  the  AIRCRAFT  MCDIFICATIONS  sub-chart,  the  number  for  each 
position  per  crew  will  be  displayed.  In  the  ORGANIZATIONAL  CHANGES 
sub-chart,  current  overhead  in  each  position  will  be  displayed.  Crew 
positions  will  be  coordinated  across  all  columns  and  scrolling 
synchronized. 

Columns  are  only  provided  for  years  in  which  changes  occur.  Only  the 
change  from  the  previous  level  is  indicated,  and  changes  are  cumulative 
across  the  columns.  (For  example,  a  change  in  CREW  is  indicated  for 
1996  and  2000.  With  BASE  indicating  a  BASE  CREW  of  1.7  per  aircraft, 
the  figure  indicates  a  force  structure  of  1.9  crew/aircraft  for  1996  and 
2.1  for  2000).  Change,  and  not  level,  is  shown  because  the  intent  of 
this  chart  is  to  indicate  what  changes  are  going  to  occur  and  when  they 
will  occur. 

No  operations  occur  under  FORCE  STRUCTURE.  It  is  provided  as  a  means  of 
representing  critical  background  information. 
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Figure  24  -  Nanning  Profile  Sub-options 


I 
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The  MflNNING  PROFILE  features  allow  the  user  to  explore  different  aspects 
of  his  problem  at  the  database  level.  He  can  look  at  the  data  and  make 
direct  correlations  with  the  people  he  knows.  He  can  look  directly  at 
short  term  losses  and  determine  how  well  his  known  gains  balance  losses. 
Support  is  provided  to  look  at  all  people  in  the  database.  Finally, 
experience  profiles  are  provided  so  the  user  has  a  feel  for  some  of  the 
factors  involved  in  the  AHACS  CAPABILITIES  assessment.  Selection  of  the 
MANNING  PROFILE  option  drops  a  pop  down  menu  of  sub-options. 

Basic  descriptions  of  the  sub-options  are  gis/en  on  as  part  of  the  help 
system,  (accessed  via  FI,  and  shown  as  the  last  storyboard).  The 
position  filter  can  be  set  from  within  the  MAMMING  PROFILE  command,  as 
indicated  on  the  menu  bar.  In  this  case,  the  filter  is  currently  set  to 
CDMT,  as  indicated  on  the  memory  aid  bar. 

One  of  the  primary  purposes  of  this  section  of  the  support  system  is  to 
provide  the  user  with  basic  data,  in  a  structured  environment,  to  allow 
him  to  exercise  his  intuition.  His  intuition  may  become  fine-tuned  in 
the  process  of  using  the  system,  or  it  may  prove  to  be  better  than  the 
models  driving  the  system,  which  should  prompt  him  to  make  hookbook 
entries  concerning  his  perception  of  system  deficiencies. 

Selecting  'Rank  filters'  causes  a  pop  down  rank  selection  area  to 
appear.  Rank  is  selected  by  clicking  and  dragging  with  a  mouse  or  by 
placing  the  cursor  on  the  desired  end  points  of  the  range  and  hitting 
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return.  (Hitting  return  twice  selects  the  single  rank  designated  by  the 
cursor).  The  ranks  displayed  in  the  selection  area  are  keyed  by  the 
selected  crew  position.  (The  rank  filter  is  the  deepest  branch  of  the 
decision  command  tree,  and  is  only  three  levels  deep). 

Rank  filtering  is  provided  to  allow  the  user  to  break  the  crew  position 
being  examined  into  smaller  chunks  that  have  special  significance  to 
him.  Rank  appears  to  be  the  best  discriminator,  since  it  is  linked  to 
respoisibil ity  level,  time  in  service,  experience,  and  special  areas  of 
qualification  within  the  crew  position. 

The  capability  of  selecting  a  range  of  ranks  is  provide  to  allow 
resolution  flexibility;  looking  at  the  top  three  may  be  sufficient  for 
some  Judgments,  while  the  crew  positions  status  with  respect  to  E-9  may 
be  critical  in  others. 
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□ 

1  POSITION 

NANNIN6  PROFILE 

RANK 

_ 

_ 

EXPERIENCE 

NAHE  FACTOR 

_ 1 _ L 

1^ 

1  1  1  I  1 

1  eta  •••  •••  ••• 

t 

i 


Help 

No^ad 

Hookbook 

Assueptions 

Prior  Screen 

Main  Nenu 

Print 

Suit 

FI 

F2 

F3 

N 

F5 

F6 

F7 

FIO 

Figure  26  -  A  Lov  Level  Data  Representation 
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This  is  a  representative  screen  for  the  GAINS  and  LOSSES  options  of  the 


MANNING  PROFILE  choice.  Both  of  these  options  require  a  crew  position 
to  be  selected,  and  are  two  of  the  three  mentioned  from  the  opening 
screen  description.  Position  can  be  changed,  but  the  "All"  option  will 
not  appear  on  the  menu. 

1 

I 

i 

This  screen  is  displayed  as  a  loss  screen.  The  memory  aid  bar  reminds 
the  user  that  he  is  looking  at  CCMTs  in  the  grades  of  E7  to  ES  who  are 

I 

projected  as  losses. 

A  scroll  bar  is  provided  to  scroll  through  the  list.  Point  and  click  I 

(mouse  or  cursor /return)  on  the  up  or  down  arrows  to  scroll  in  the 
indicated  direction,  with  iite  current  position  in  the  list  being 
indicated  by  the  left  pointing  arrow.  (As  in  Macintosh  operations). 

The  sub^hart  heading  will  remain  to  stationary  during  scrolling.  The 
roster  is  indexed  on  RA(nK  then  LOSS  DATE  and  then  NAME.  An  indexing 
command  can  be  supported  and  implemented  through  a  pop-up  priority 
selection  table. 
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Appendix  C 


Hookbook  Entries 

This  appendix  contains  the  aggregated  contents  of  the  hookbook 
entries  made  during  the  research  effort.  The  IDEA  portions  of  the 
individual  hookbook  entries  have  been  grouped  by  the  system’s  adaptation 
areas  that  were  evident  to  the  author. 
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RESOLUTION 

RESOLUTION 

RESOLUTION 

RESOLUTION 

MODELS  ~ 

MODELS  — 

MODELS  — 

MODELS  — 

MODELS  -- 

MODELS  — 

MODELS  — 


DON'T  BECOhE  CONSUED  WITH  BUILDING  HIGH  RESOLUTION 
MODELS.  CERTAINLY,  AT  THE  BEGINMING,  LOOKING  AT  NEEDS 
BASE  ON  AN  TIME  FRAIE  IS  ADEQUATE. 

SUPPORT  CmvJGES  IN  THE  CURRENT  YEAR  RESULTING  FROM 
QUWTERLY  UPDATED  MEETINGS. 

START  LOOKING  AT  TRAINING  FLOW  AND  HE  PROBLEMS  IT 
INTRODUCES  IN  THE  SYSTEM.  TVESE  TYPES  OF  FACTORS  WILL 
FOR  fiN  INCREASE  IN  RESOLUTION,  SINCE  TRAINING  ISN'T 
GENERALLY  UNIFORM  ACROSS  A  YB«. 

SPECIFICATIONS  INITIALLY,  WITH  fGSUt^riONB  THAT 
TRAINING  IS  UNIFORM  ACROSS  THE  YEAR  (NOT  TRUE,  BUT  IN  THE 
SCOPE  OF  THIS  PROTLEM  AT  THE  INITIAL  STAGES,  H-E 
VARIATIONS  IN  MUMMING  AND  EXPERIENCE  CAUSED  BY  NON- 
UNIFORM  TRAINING  TVROUGH  THE  YBR  fiRE  INSIGNIFICANT  WHEN 
COMPARED  TO  THE  PROBLEMS  CAUSED  BY  MANAGERS  NOT  LOOKING 
AT  THE  ACQUISITION  PROCESS  IN  ANY  DETAIL  BEYOlvff)  THE 
irtEDIATE  YEAR'S  ^EEDS. 

THREE  BIGGIES  —  EXPERIENCE  —  MWMMING  LEVEL  — 
CAPABILITIES 

EACH  OF  THE  TFREE  MAJOR  MODEL  HAVE  TO  BE  ABLE  TO  LOOK  AT 
"NOW"  WD  PROJECT  TO  THE  FUTURE 

IN  ORDER  TO  MA<E  PROJECTIONS,  TPE  MODELS  (RE  GOING  TO 
HAVE  TO  INTERACT 

PERSONNEL  ISSUES  (RE  "SOFT"  ISSUES.  IF  MCDM 
f'ETHODCl.OGIES  ARE  USED  IN  THE  MODEL  DEVELOPfENT,  TVE 
MODELS  MAY  BENEFIT  FROM  A  STATISTICAL  VALIDATION  OF  THE 
VALUES  THEY  PRODUCE. 

THE  CAPABILITIES  MODEL  CAN  CERTAIN_Y  BE  EXPRESSED  AS  A 
MULTI-CRITERIA  OPTIMIZATION  PROBLEM  (OR  SOME  OTHER  FORM 
OF  MCDM  MODEL).  IT  IS  CONCER^ED  WITH  BM_ANCING 
CONFLICTING  OBJECTIVES  —  MANMING  LEVEL  PM)  EXPERIENCE, 
FOR  EACH  CREW  FORCE  —  WHERE  ARE  RESOURCES  TO  BE 
ALLOCATED. 

CAPABILITIES  NEEDS  TO  BE  ASSESSED  A(»INST  MULTIPLE 
EXPERIENCE  CRITERIA.  MEAN  EXPERIENCE  IS  NOT  SUFFICIENT, 
SOME  DISTRIBUTION  MEASURES  WILL  ALSO  BE  NEEDED. 

OPPS!  !  SCHNEIDER'S  DEPARTURE  MODELS  ARE  NOT  GOING  TO  BE 
ADEQUATE.  DEPARTURES  (RE  GOING  TO  HAVE  TO  BE  BROKEN  OUT 
INTO  CATEGORIES  THAT  CAN  BE  CORRELATED  TO  EXPERIENCE  IF 
EXPERIENCE  PROJECTIONS  (RE  GOING  TO  BE  M(«E. 
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MODELS  — 


MODELS  — 

MODELS  — 

MODELS  — 

MODELS  -- 

MODELS  — 

MODELS  -- 


MODELS  — 


ASSUr^PTIONS:  PROGRESSION  IN  EXPERIENCE  WILL  BE  BASED  ON 
ALL  PER90MVEL  IN  THE  POSITION  DOING  IHINQS  THROUGH  TIME, 
SUCH  AS  l-EETING  THEIR  MINIMUM  QLJ«_IFICATION  FLYING 
HOUR/MISSIONS  QOA_S.  EXCESS  FLYING  HOURS  WILL  BE 

APPORTIOtvED  UNIFORMLY  FV<t3MG  THE  CREW  FORCE  IN  THIS  CASE. 
THIS  WOULD  BE  A  MCCELING  ASSUMPTION,  AW)  HEEDS  TO  BE 
STATED  AS  SUCH. 

ASSUMPTIONS;  MODELS  MAKE  LOTS  OF  ASSUMPTIONS  —  DOCUMENT 
THEM. 

MODELS  ARE  GOING  TO  HAWE  TO  HANDLE  LOTS  OF  TRICK 
QUESTIONS  TO  MM<E  PROJECTIONS.  TR^vEITIONS  ACROSS  CLASS 
BOUNDARIES  WILL  FPWE  TO  BE  mMDLED  WD  DOCUMENTED. 

DISCRIMIWWT  FJ^ALYSIS  BEARS  EXAMINATION  IN  THE  FUTURE, 

BUT  NOT  NOW.  UNTIL  THE  SYSTEM  IS  CHANGED  ENOUGH  THAT 
THERE  IS  SOME  PROB^eiLITY  OF  RECOVERING  PERSOWEL  FROM 
THE  AIR  FORCE  fPMMING  POOL,  PRIORS  FOR  THE  DISCRIMINWT 
FUNCTION  C«MvlOT  BE  ESTIMATED.  (ACTUALLY  0  CURRENTLY, 
WHICH  DOESN'T  HELP  DISCRIMI^TE) . 

ARTIFICIAL  INTELLIGENCE  MODELS  ARE  NOT  A  PLAYER  IN  THIS 
SYSTEM  UNTIL  IT,  FM)  THE  USERS  MATURE  QUITE  A  BIT.  AI  IS 
A  POSSIBILITY  AT  SOME  FUTURE  DATE  FOR  SPEEDING  UP  THE 
"WHAT-IF"  CYCLE  IN  THE  SCHEDULING  OPTION. 

A  GOOD  EXPERIENCE  MODEL  ~  PROBABILITY  OF  BEING  rtM 
INSTRUCTOR  /  STAN  EVALER.  ASSUMPTION  —  THESE  PEOPLE  PFE 
AT  THE  HIGH  END  OF  THE  EXPERTISE  SCALE  FOR  THE  CREW 
POSITION. 

QUESTIONS  —  ANSWERS  REQUIRE  STATEMENT  OF  ASSUMPTIONS 

1.  WHAT  PPE  THE  MECHWMISMS  FOR  PREDICTING  LOSSES  IN  EACH 

CREW  POSITION 

-  RATES  BY  R«M<  —  SEPARATIONS  PND  LOSSES  —  BASED 
ON  AWACS  DATA  —  OR  IS  AIR  FORCE  WIDE  DATA 
SUFFICIENT  UNTIL  AWACS  DATA  IS  DEVELOPED? 

-  RATES  BY  YEARS  OF  SERVICE  ?? 

-  IMPACT  OF  THINGS  LIKE  FORCE  REDUCTION 

2.  SAME  FOR  GAINS  ~  PROJECTING  PROMOTIONS  IS  IMPORTANT 
IF  LOBS  RATES  PfE  BY  GRADE. 

LOOK  AT  THE  RESULTS'  !  JUST  BECAUSE  THE  MODELS  SAY 
SOMETHING,  DON'T  TA<E  IT  AT  FACE  V«_UE.  USING 
INSTRUCTORS  AS  AN  EXPERT  BASELINE  COULD  REALLY  GIVE 
SCREWY  RESULTS  IN  A  REGRESSION  IF  A)  BEING  AN  INSTRUCTOR 
IS  VIEWED  AS  A  PROMOTION  POSITION,  AND  B)  YOU  DON'T  CAP 
FACTORS  SUCH  AS  YEARS  OF  SERVICE  FOR  PASSED  OVER 
OFFICERS. 
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MODELS  — 


MODELS 


MODELS  -- 

MODELS  — 

MODELS  — 

MODELS  — 

EXTENSIONS 


m<E  MODELS  ABLE  TO  RECON1END  OPTIMUM  STRATEGIES,  THEN 
RESPOND  TO  ■■WWT-IFS"  FROM  T>^T  INITIAL  ST^«TING  POINT  ■;> 
DO  I  REALLY  WWT  TO  IMPINGE  ON  1>E  USERS  UmT  IF 
EXPLORATION  ?  HOW  DO  KEY  THE  USER  TO  START  HIS 
PROCESS?  THIS  AT  LEAST  GIVES  HIM  A  ST^^ING  POINT 
BESIDES  A  BLArfvK  SCREEN. 

POSSIBLE  P^«rt-ETERS  FOR  SOEDULING  SCREEN 
CCtt)E  55 
AVG  ON  STATION 
ACCESSION  SOURCES 
EDUCATIONS 
TRAINING 
FLYING  HOURS 
NEW/USED  SPLITS 

Tl-ESE  ARE  JUST  ODD  THOUGHTS  ON  POSSIBLE  PPRf^lETERS  TmT 
LEM)  THEMSEL'«€S  TO  MANIPULATION  BY  THE  USER  TO  «5RIVE  AT 
A  DECISION.  THESE  ARE  THE  PARW'tTERS  TmT  VrtRY  TT-E 
MODELS  OUTPUTS.  FINfl.  DETERMimTIGN  OF  f«=PROPRIATE 
PARW1ETERS  WILL  HAVE  TO  BE  BASED  ON  THE  FACTORS  TmT  GO 
INTO  GmvfTIFYING  EXPERIENCE.  THE  DATA  IS  GOING  TO  DECIDE 
THIS.  PARAMETERS  WILL  HAVE  TO  HAVE  SOME  DIRECT 
CORRELATION  TO  THE  INPUTS  TO  THE  MODELS. 

OTHER  PARAMETERS  —  CONCEIVABLE  MORE  IMPORTANT  THAN  MAJOR 
PARAETERS  ABOVE  BUT  D-EY  WOULD  REQUIRE  POLICY  CHANGES 
OUTSIDE  OF  ThE  r*¥NAGERS  SCOPE  OF  OPERATION 
CODE  55 

AVG  TOUR  LENGTH 
ACCESSION  SOURCE 
MISSIONS  FLOWN 

DON'T  FORGET  DOS  ETC  AS  INPUTS  TO  LOSS  CALCULATIONS  fWE 

SURE  ALL  DRIVERS  FOR  miNS  AND  LOSSES  ARE 

CONSIDERED. 

mJOR  PARAIETERS  TO  BE  MANIPULATED  BY  CURRENT  MANAGER  — 
BODIES  fiND  TIMING. 

PRINCIPLE  CaM=ONENTS  ANALYSIS  COULD  hELP  SCREEN  THE 
POSSIBLE  FACTORS  TO  BE  CONSIDERED  IN  THE  EXPERIENCE 
ANALYSIS.  THIS  COULD  SIGNIFICANTLY  SIMPLIFY  DATABASE 
ACCESS  ISSUES. 

GmPHS  PROVIDE  QUANTITATIVE  AWD  IN  BARGAINING  WITH 
MPC/TAC  CONSEQUENCES  ARE  IMMEDIATELY  DISCERNABLE.  OCCURS 
TO  ^E  TmT  THESE  GRAPHICAL  PRESENTATIONS  CAN  PROBABLY  BE 
UNDERSTOCK)  BY  TAC  AM5  AFMPC  ~  GOOD  AVMD.  THIS  C:AN  ALSO 
PROBABLY  BE  CONSIDERED  A  CONTROL  l«ECmNISM. 
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EXTENSIONS  — 

EXTENSIONS  — 

EXTENSIONS  ~ 

PRESENTATION  ■ 

PRESElvrrATION  ■ 

PRESENTATION  • 

PRESENTATION  ■ 

PRESENTATION  • 

PRESENTATION  ■ 


EXTENSION  OF  THE  USE  OF  THIS  SYSTEM  WOULD  BE  TO 
IDENTIFY  INDIVIDU^S  UHO  fiRE  "BEHIND  THE  CURVE"  AND  FLY 
THEM  MORE  OFTEN. 

THERE  IS  A  Di^vJQER  THAT  THIS  SYSTEM  COULD  BECOME  A  SQUARE 
TO  BE  FILLED.  HOW  TO  AVOID  THIS,  IF  IT  IS  SOMETHING  TO 
BE  AVOIDED.  IF  FILLING  THE  SQU^^ES  BECOHES  THE  GOAL,  «V1D 
IT  IHPRQVES  "EXPERIENCE",  IS  THIS  A  PROBLEM.  THE 
VULNERABILITY  OF  THE  SYSTEM  BEING  »¥ED  FOR  OTHER 
PURPOSES  IS  VERY  DEPSnIDENT  ON  THE  FACTORS  CHOSEN  TO 
INDICATE  EXPERIENCE.  FOR  FACTORS  SUCH  AS  TIME  IN 
SERVICE,  THERE  IS  NOTHING  THE  MEMBER  DO  ABOUT  ITS 
IMPACT  ON  HIS  EXPBRIENCE  SCORE. 

EXPERIENCE  SCORES  COULD  BE  USED  TO  HELP  IDENTIFY 
C^MDIDATES  FOR  "SPECIAL  JOBS"  SUCH  AS  INSTRUCTOR,  STPN 
EVAL  OR  SCHOOL  ATTENDANCE.  THIS  COULD  ALSO  BE  USED  AS  A 
MODEL  VALIDATION  TOOL,  SINCE  CONSISTENT  DISAGREEMENT  HWY 
INDICATE  EXPERIENCE  IS  BEING  SCORED  WRONG. 

SPLIT  SCREEN  FOR  CH^TS  —  ENCOURAGE  DIRECT  COHPW^ISONS, 
DON'T  HAVE  TO  GO  TO  PRINTER  IF  POSSIBLE  —  PARTICULARLY 
FOR  EXPERIENCE  AND  MANNING 

TOGGLE  ON  YEAR  TO  BLOWUP  PROFILE  IN  1  YEAR  PERIODS  POP  UP 
TABLE  OF  MIN/rWX  VALUES,  AVG,  STD,  OTHER  PERTINENT  DATA 
TO  CLARIFY/SUPPORT  GRAPHICS  DISPLAYS.  CLOSE  LOOKS  AT 
DATA  WILL  HAVE  TO  BE  SUPPORTED,  AND  INTERPOLATION 
PROBLEMS  SHOULD  BE  AVOIDED. 

ADD  TABLES  OF  CRITICAL  INTW1ATION  KEYED  TO  GRAPHS. 

HOW  TO  DISPLAY  EXPERIENCE  DISTRIBUTION  ON  PROJECTIONS  ~ 
—  HAVE  A  BAND  INSTEAD)  OF  A  LINE  TO  ACCOUNT  FOR  VAR  AfiOUT 
AVERAGE  VALUE  ?? 

PERWPS  A  COLOR  QRfiDlENT  IN  THE  LINE  IS  MORE  EFFECTIVE 
THAN  A  BAND.  MAYBE  A  SII*PLE  GREEN  TO  YELLOW  TO  RED  TO 
SIGNAL  STRUCTURAL  HEACTH  OF  THE  CREW  PROFILE 

DISPLAY  EXPERIENCE  AS  A  HISTOGRW  —  PROJECTIONS  BUILD 
HISTOGRANS  THAT  CAN  BE  COHPARED  ACROSS  YEARS 

IF  HECESSARY,  INCREA^  THE  VERTICAL  RESOLUTION  ON  THE 
HISTOGRANS  BY  STARTING  THE  SCALE  AT  SOME  VA^UE  OTHER  THWM 
0.  FOR  THE  FEW  BARS  THAT  ARE  SHORTER  THWM  THE  BA«E 
VA^UE,  WRITE  THE  MARGINAL  AMOUNT  ABOVE  THE  BAR. 
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PRESENTATION  —A  POSSIBLE  REFINEMENT  OF  THE  EXPERIENCE  LEVEL 

PRESENTATION  IS  A  GO-ND  GO  INDICATOR  FOR  THE  DISTRIBUTION 
P/=R(=METERS  (KIRTOSIS  AND  SKEWNESS  IN  PARTICULi«)  THAT 
INDICATES  WhCn-ER  TEE  CREW  PROFILE  IN  A  GIVEN  YEAR  IS 
GOOD.  FOR  THE  CURRENT  WORK,  A  GRAPHICAL  LOOK  AT  THE 
DISTRIBUTION  IS  ALL  THAT  WILL  BE  PRESENTED. 

PRESENTATION  —SUPPLYING  REFERENCES  FOR  THE  MODELING  RESULTS  TO  BE 

hEASURE  AGAINST  CAN  BE  TRICKY.  FDR  BODIES,  IT  IS  EASY 
SINCE  THE  REFERENCE  IS  JUST  THE  AUTHORIZED  mMMING  LEVEL. 
FOR  EXPERIENCE,  THE  REFERENCE  WILL  HAVE  TO  BE  BUILT  SOTE 
HOW  —  EXPERT  OPINION  (HADA)  —  USE  AUTHORIZED  RA^K 
STRUCTURE,  ASSUMING  THAT  RANK  STRUCTURE  IS  HOW  MPC  TRIES 
TO  MAIV^VQE  EXPERIENCE  7T> 

PRESENTATION  -  rt_LOW  Tl-E  USER  TO  EXAMINE  HIS  "WhkAT-IFS"  ON  THE  SAME 
SCREEN.  USE  MULTIPLE  WINDOWS  FOR  THE  USER  TO  LOAD 
DIFFERENT  PARA^tTBR  VALUES  INTO,  AND  DRAW  THE  LINE  FOR 
EACH  WINDOW  ON  A  COMMON  GRAfU.  COLOR  CODE  LINES  ON  THE 
GRAPH  WITH  THE  WIfvOOW  IT  CORRESPONDS  TO.  THIS  ALLOWS  THE 
USER  TO  KEEP  CLEAR  WHAT  HE  HAS  CONSIDERED  ^  HOW  THE 
RESULTS  COMPARE. 

DESIGN  —  THE  SCHEDULING  PROCESS  REQUIRES  MORE  THAN  A  SII-PLE 

CONSIDERATION  OF  MANNING  LEVEL  IN  TERMS  OF  BODIES. 
EXPERIENCE  H¥«  TO  BE  OBSERVABLE  AT  THE  SA^  TIME. 

DESIGN  —  PANACIGM  —  HATCH  LINES  FOR  DECISION  -  PRE-AFIT 

THOUGHT  ON  HOW  TO  BEST  lAKE  THE  DECISION  BEING  SUPPORTED. 
KEEP  IT  SIHPLE!  !  !  THIS  ENTRY  ALSO  PROPERLY  FALLS  UtvflDER 
OPERATION. 

DESIGN  —  PARAADIGM  IS  CRITICAL 

DESIGN  —  DESIGN  TOOLS  APPEAR  TO  BE  IN  SHORT  (NON-EXISTENT  ?) 

SUPPLY.  WHY?  THEY  ARE  NEEDED,  SO  WHY  AREN’T  THEY 
AVAILAABLE. 

DESIGN  —  STORYBOARDING  IN  A  WORD  PROCESSOR  IS  A  HORRIBLE 

EXPERIENCE  AAGWIN,  WHY  AREN’T  THERE  GOOD  TOOLS. 
STORYBOARDING  TOOLS  THAT  BUILD  INTERACTIVE  STORYBOARDS 
SHOULD  BE  AAVAILAABLE  !  ' 

DESIGN  —  DSS  IS  DEAD  IF  SOMEBODY  DOESN’T  BUILD  AN  EFFECTIVE  DSS 

GEIvERATOR.  WRITING  CODE  FROM  SCRATCH  TAKES  TO  LOMG. 
WITHOUT  A  GEHERATOR,  AN  OUTSTANDING  SET  OF  TOOLS  WILL 
HAVE  TO  BE  BUILT  IN  AADVANCE  OF  ATTEMPTING  TO  DESIGN  A 


DESIGN  —  REGARDLESS  OF  HOW  HRD  IT  IS  TO  CODE,  SPREADSHEET 

ENVIROHMENTS  ARE  A  STOP  QfP  AT  BEST.  mCROS  ARE 
INSUFFICIENT,  AND  BLOCK  OPERATIONS  ON  DATABASE  ELEMENTS 
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fiRE.  DIFFICULT  SINCE  A)  FIELD  NAI-ES  MUST  BE  KNOW,  B) 
OPERATIONS  PRE  GENERALLY  DIFFICULT  TO  DEFINE  UNLESS  TT-EY 
PRE  l-m)WIRED  INTO  1>C  SPREADSHEET. 


DESIGN  —  QUATTRQ  PRO  mV  BE  CLOSE  TO  A  FULLY  ALJTCrV>TED 

STORYBQP«DING  TOOL 

DESIGN  —  DON'T  FORGET  THAT  MANNING  IS  CYCLIC  --  PAY  SPECIAL 

ATTENTION  TO  PERIODS  OF  HIGH  ACQUISITION  DEmJD;  D«*P 
FLUCTUATIONS  —  IF  207.  OF  THE  CREW  FORCE  NEEDS  TO  BE 
REPLACED  THIS  YEAR,  THE  SPy*E  WILL  BE  TRUE  IN  ROUGHLY  FIVE 
YEPRS  WHEN  ALL  OF  THE  NEW  BODIES  PCS.  THIS  "BOOM  OR 
BUST"  CYCLE  MUST  BE  BROUGHT  UNDER  CONTTRCL.  CURRENTLY, 

THE  CYCLIC  iv^TURE  OF  rWsIvilNG  IS  IGNORED  BECAUSE  OF  THE 
SHORT  TERM  ORIENTATION  OF  THE  PLAMvJING  PROCESS. 

DESIGN  --  DISPLAY  -  1  YEAR  +  5  SCHEDULING  GRAPH,  NOT  10  WEfiRS  AS 

ORIGIlvWLLY  CONCEIVED  --  THE  CHAIN  OF  ASSUT^TIONS  IS  GOING 
TO  BE  EXTENDED  TO  LONG  FOR  RESULTS  TO  HAVE  fiNY  rEPWING  IN 
THE  10  YEPR  TIME  FRP^E. 

CONTROL  —  SELECTION  OF  POSITION  IN  SYSTEM  RESPONSIVE  TO  FIRST 

LETTER  STRING  FULL  BLOWN  COTTIAND  LPNGUAGE  TO  COHPLEX  FOR 
US  TO  ACHIEVE 

CONTROL  —  AUTO  ENTRY  IN  HOOK  OF  SCREEN  FROM  WHICH  ENTRY  IS  HADE 
AUTO  ENTRY  IN  HOOK  OF  PROGRESSION  TO  SCREEN 
DATE/TIHE/OPERATOR  ST/W  IN  HOOK  ENTRIES 

CONTROL  —  ASSUMPTIONS  MUST  BE  VISIBLE  IF  USER  IS  TO  SUCCESSFULLY 

HAVIGATE  THROUGH  A  DECISION  PROCESS  —  ADD  PM  ASSUMPTIONS 
BUTTON  AS  IN  EL  SHERIF 

IMPLEMENTATION-WHAT  IS  NEEDED  IS  AN  ENVIR0M1ENT  THAT  FOSTERS  DEVELOPHENT 
OF  PM  OPERATIOtvAL  SYSTEM,  AND  THEN  GENERATES  THE  SPECIFIC 
STAND  P^OHE  CODE  TO  EXECUTE  ALL  OF  THE  FUNCTIONS  THAT 
HAVE  BEEN  DEVELOPED. 

UNTIL  THEN,  BUILD  PROTOTYPES  IN  SOME  ENVIROMENT  THAT 
PROVIDES  ALL  OF  THE  NECESSARY  FUNCTIONS,  PND  THEN 
REPROGRAM,  EVEN  THOUGH  THIS  EFFECTIVELY  LOOSES  THE 
DEVELOPER  A  HAJOR  PART  OF  THE  EFFORT  DEVOTED  TO 
PROTOTYPING. 

IMPLEMENTATION-REGP«DLESS  OF  WHEJ^  A  DSS  IS  DESIGNED,  THE  IMPLEMENTED 
VERSION  HEEDS  TO  BE  IN  STAND  Pt.OHE  CODE  FOR  PERFORHAnJCE 
PND  FLEXIBILITY  —  WHERE  IS  MY  ITS  GENERATOR? 

VALIDATION  —  THE  PROCESS  OF  REPEATEDLY  OPERATING  THE  MODELS  WITH 

VPRIED  PPfWtTERS  WILL  PROVIDE  INSTRUCTION  IN  THE  SYSTEMS 
USE  AND  VP^IDATION  FOR  THE  MODELS.  OBSERVABLE 
CONSEQUENCES  OF  PARAMETER  CHm3ES  CAN  BE  CHECKED  AGAINST 
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EVOLLmON  — 

EVOLUTION  — 

EVOLUTION  — 

DATA  — 


THE  l'¥M¥3ER»S  INTUITIONS,  WHICH  WILL  I^TURE  AS  ThE  SYSTEM 
IS  EXERCISED. 

IT  MUST  BE  t-ADE  CLEPR  TO  1TE  USERS  imJ  Tl-E  SYSTEM  WILL 
ONLY  BE  AS  GOOD  AS  THEY  m<E  IT.  USERS  MUST  KNOW  TmT 
ALL  USER  INPUTS  ARE  INPORTAMT,  PND  MORE,  THAT  AT  A 
MINIMUM,  USER  INPUT  IS  NECESSARY  TO  FINE  TUNE  THE  MODELS 
USED  TO  SUPPORT  THEIR  DECISIONS. 

AUTO  ENTRY  IN  HOOK  OF  SCREEN  FROM  WHICH  ENTRY  IS  h'ADE 
AUTO  ENTRY  IN  HOOK  OF  PROGRESSION  TO  SCREEN 
DATE/TIME/OPERATOR  STAMP  IN  HOOK  ENTRIES 

ASSUNPTIONS  MUST  BE  VISIBLE  FDR  EVOLUTION  TO  OCCUR  AMD 
PN  ASSUMPTIONS  BUTTON  AS  IN  EL  SHERIF 

POSSIBLE  FACTORS  -  JUST  A  DUHP 

FORCE  STRUCTURE 

-  OVERHEAD/ST/2fT-SOF-DETCO-ETC 
<  PPA/SCHEDULED  CHANGES) 

(  MOD/SOCDULED  UPGRADES) 

(  CREW  COMPOSITION  BY  MOD) 

NANNING  LEVEL  -  BODIES  VS  AUTHORIZATIONS 

-  RPN< 

-  PFSC 

-  SCHEDULING  ACQUISITIONS 

—  WHO’S  LEAVING 

(CODE  55,  TOS,  AVG  TOUR  LENGTH,  LOSS  OR 
"PCA") 

--  NEW  REQUIREMENTS 

(NEW  PFT,  COLLATERAL  ACCESSION,  RECAPTURE) 
(NEW  REQUIREMENTS  FROM  FORCE  STRUCTURE  / 
BUILD-UP  ISSUES) 

-  PROFILES  (PROJECTIONS/CONSEQUENCES) 

—  TABLES  AMD  LINE 

EXPERIENCE  LEVELS  -  FULL  UP  CREWS 

-  RANK 

-  AFSC 

-  LOSSES 

-  MODELS  -  RECONTENDATIONS  OR  RESPONSE 

-  MODEL  (ALL  ELEMENTS  DATABASE  INPUTS  TO  MODEL/ 

ALSO  POSSIBLE  PARAMETERS) 

-  TRAINING 

-  EDUCATION 

-  YEARS  AWACS 

-  #  MISSIONS 

-  #  HOURS  FLOWN 

-  #  INTERCEPTS 

-  YEARS  COLLATERAL  EXPERIENCE 
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-  EF?RaR  HISTORIES 

--  Tf^ 

—  LOGS 

--  miNT  IM=UT  (E.G. ,  CND) 

-  PROFILES (PROJECT IONS) /CONSEQUENCES 

—  TABLES  AND  LIlvE 
SYSTEM  a¥>ABILITIES/PERSONNEL  IMPACT 

-  MODEL 

(EFFECTIVE  INTERCEPTS)  (REGRESSIONS) 
(CONTINUOUS  mRDWARE  OPS) 

(CX3  BREAKDOWNS) 

-  PROFILES (PROJECTIONS) /CONSEQUENCES 

--  TABLES  AND  LINE  REPRESENTATIONS 

DATA  ~  DON'T  FORGET  DOS  ETC  AS  INPUTS  TO  LOSS  CALCULATIONS  I'WE 

SURE  ALL  DRIVERS  FOR  GAINS  AND  LOSSES  ARE 
CONSIDERED. 
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Appendix  D 


Data  Requirements 

This  appendix  lists  the  types  of  personnel  data  that  can  be  useful 
for  each  of  the  major  models  needed  to  make  the  DSS  function.  The  list 
is  not  comprehensive;  only  extensive  "data  snooping"  at  the  user  site 
can  ensure  all  useful  data  is  identified. 

Data  elements  suggested  for  examination  are  listed  with  the 
location  of  the  data.  The  28  AD  database  mentioned  is  not  currently  in 
place,  although  it  has  been  fully  developed.  Until  such  time  as  this 
database  is  fielded,  the  needed  data  can  be  found  in  the  squadron  ORCs. 

As  a  general  point,  where  data  elements  pertaining  to  time  are 
concerned,  the  databases  will  often  contain  originating  dates  rather 
than  cumulative  time.  Arriving  at  the  necessary  data  will  require 
subtracting  the  originating  date  from  the  current  date. 
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DATA  BEARING  ON  rWylNJING  LEVEL 


SOURCE 

! 

28AD  j 

A-FORMS  CBPO 

DB  1 

j 

FORCE  STRUCTURE  CHWMGES 

1 

i 

1. 

E-3  MODIFICATIONS 

CURRENTLY,  NOTE  OF 

THIS  j 

2. 

OR»WIZATIOm_  OWNGES 

INF0R1V>TI0N  IS  AUTOmTED.  IT  ! 

3. 

PAA  OMVJGES 

DOES  EXIST  AT  TLE  DIVISION  j 

4. 

CREWS/FW>  CHANGES 

AT  28  AD/XO. 

i 

LOSS 

FACTORS 

5. 

A\,€RAGE  TIME  IN  AWACS 

X 

X 

6. 

AVERAGE  TIME  AT  TINKER 

X 

X 

7. 

CODE  55  LENGTH 

X 

e. 

RfiN< 

X 

X 

9. 

PLi^MvED  RETIREMENT  DATES 

X 

X 

10. 

PLPMSED  SEPARATIONS  DATES 

X 

X  i 

11. 

SOEDULED  PCS  DATES 

X 

X 

12. 

LOSS  RATES 

i 

BY  TINE  IN  SERVICE 

MPC  CURRENTLY 

X 

BY  RANK 

II  M 

X 

BY  CREW  fiFSC 

H  H 

X 

BY  #  OF  FLYING  HOURS 

X 

X  ! 

13. 

NUMBER  OF  REMOTES 

X 

X 

TRAINING  LIMITATIONS 

12. 

CAPACITY 

552  TTS 

PERSONMEL  DATA  POSSIBLY 

BEARING  ON  EXPERIENCE 

SOURCE 

2GA0 

A-FORMS 

CBPO 

DB 

FLYING  HISTORY 

1. 

TOTAL  YEARS 

X 

2. 

E-3  YEARS 

X 

3. 

TOTAL  HOURS 

X 

4. 

E-3  HOURS 

X 

SERVICE  AMD  ASSIGIMENT  SPECIFICS 

5. 

YEARS  AMACS 

CONUS 

X 

X 

NWTO,  PACAR,  ICELAND 

X 

X 

6. 

NUMBER  SHORT  TOURS 

X 

X 

7. 

NUMBER  LONG  TOURS 

X 

X 

8. 

YEARS  CURRENT  CREW  POSITION 

X 

X 

9. 

YEARS  OTHER  AWACS  POSITIONS 

X 

X 

10. 

MAX  STARF  LEVEL 

X 

X 
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SQ  (1);  WING  (2); 

DIVISION  (3);  HDS  (4) 

11.  YEARS  IN  POSITIONS 

12.  APPLICABLE  NON-AWACS  EXPERIENCE 

YEARS  GROUND  TACCs  <WD,  SD,  AST) 
YEj=«S  C3  (FOR  MCC,  SD) 

ABCCC  X 

RIVIT  JOINT  X 

COr-PASS  C^^  X 

ETC. 

13.  YEARS  IN  SECONDARY  RELATED  ATSC  X 

(RAD«?,  COMPUTERS) 

14.  GRADE 

15.  TIME  IN  SERVICE 

16.  TIME  IN  GRADE 


X  X 


MISSION  HISTORY 

17.  #  SORTIES  X 

18.  #  MISSIONS  X 

19.  #  EXERCISE  MISSIONS  X 

20.  #  TEST  MISSIONS/EVAL  MISSIONS  X 

21.  4  SIMULATOR  SESSIONS  X 

22.  #  OF  CONTROLS/ INTERCEPTS 

(SD,  WD,  ASO,  AST)  X 


X 

X 

X 

X 

X 

X 


TRAINING  WD  PERFORmNCE  SPECIFICS 

23.  FINAL  TRAINING  EVALUATION  X 

24.  ATTEMmiE  OF  SPECIE.  SCHOOLS 

FWS,  TACATC  (WD,  SD,  AST)  X 

TYPE  2  TRAINING  (CDMT,  CT,  CSO,  ART)  X 

25.  ST^vEVAL  RATINGS  X 

26.  YE#«S  SPECIAL  POSITION  (^T) 

27.  APR/OER  X 


X 

X 

X 

X 

X 
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X  X 


flPPEM)IX  E 


Model  Requirements 

A  description  of  the  requireoents  for  the  three  major  models  are 
included  in  this  appendix.  The  descriptions  for  the  manning  level 
models  and  the  capabilities  models  are  very  general.  The  modeling 
methodology  for  the  kernel  is  discussed  more  fully. 
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Mamina  Level  ftodels 

The  manning  level  models  drive  the  scheduling  displays,  and  show 
the  projected  manning  status  of  the  crew  positions.  They  must  1) 
project  losses  from  the  crew  position,  and  2)  roll  the  projected 
acquisitions  described  by  the  manager  during  his  "uhat-if"  sessions  back 
into  the  crew  position  to  arrive  at  a  manning  level  projection  across  a 
five  year  PFT  planning  period. 

There  are  multiple  available  strategies  available  for  projecting 
losses  (and  this  is  obviously  the  hard  part  of  the  model).  Far  example, 
loss  rates  developed  from  historical  data  can  be  used.  This  can  come 
from  AUlPCS  data,  or,  if  that  data  is  not  currently  available,  data 
may  be  available  by  general  AFSC  (to  be  used  until  AUIACS  specific 
historical  data  is  collected).  Another  possibility  is  to  use  average 
time  in  AUACS  estimate  identify  individuals  who  are  likely  to  leave. 

The  second  method  is  probably  more  complex  to  implement,  but  should  give 
truer  results  since  it  is  know  that  loss  rates  are  not  constant  across 
the  years. 

However  losses  are  projected,  care  must  be  taken  to  make  individual 
projections  for  distinctly  different  groups,  and  then  aggregate  these 
losses  for  an  annual  loss  picture.  As  an  example,  first  time  AWAC^s 
accrue  a  service  commitment  (in  the  form  of  a  code  55)  that  locks  them 
into  the  AWACS  system  for  some  period  of  time.  People  returning  to 
AkJACS  from  other  assignments  do  not  have  this  commitment.  Therefore, 
there  may  be  a  significant  difference  in  the  average  time  in  AWACB  for 
these  two  groups  before  they  become  lost  to  the  AWACS  manning  pool. 

Care  must  also  be  taken  to  project  losses  by  personnel  groupings 
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that  are  distinctly  different  in  experience,  so  that  the  experience 
level  models  can  make  experience  level  projections.  For  example,  if  a 
major  factor  in  experience  level  is  rank,  projected  losses  could  be 
broken  out  by  rank.  Identifying  losses  by  experience  categories  will 
allow  the  experience  models  to  assess  losses  in  experience  do  to 
departures  from  AUJACS. 

Experience  Level  Mcxiels 

The  experience  level  models  need  to  accomplish  three  distinct 
functions.  First,  they  need  to  assess  experience  for  each  of  the 
individuals  in  a  crew  position.  This  portion  forms  the  kernel  of  the 
DSS  presented  here,  and  is  discussed  more  extensively  below. 

The  second  function  the  model  must  perform  is  to  manage  char>qes  in 
factor  levels  u*iile  projections  are  being  made.  (How  these  changes  are 
managed  is  a  key  area  of  system  assumptions  that  must  be  documented). 

As  an  example,  if  flying  hours  are  significant  in  assessing  experience, 
a  distribution  scheme  for  allocating  flying  hours  beyond  bare 
qualifications  necessities  is  required.  Obviously,  the  easy  approach  is 
to  average  the  total  flying  hours  available  over  the  entire  crew  force. 
The  question  is,  is  this  v/iat  actually  happens,  and  if  not,  how 
different  are  the  experience  profiles  developed  using  averages  from 
those  using  a  more  difficult  allocation  scheme  (over  a  five  year  period, 
this  may  be  a  moot  point). 

Finally,  the  model  must  make  projections  for  experience  levels 
across  a  five  year  period,  using  the  factors  discussed  in  the  above  two 
paragraphs,  and  the  information  of  losses  by  category  that  the  manning 
level  model  must  provide. 
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With  this  general  overview  of  the  experience  level  model 
requirements,  the  rest  of  the  discussion  relative  to  experience  modeling 
presented  here  pertains  to  the  experience  scoring  method  developed  for 
the  kernel  DSS. 

Experience  Scoring.  As  discussed  in  chapters  two  and  four,  the 
approach  adopted  in  the  kernel  DSS  for  developing  experience  scores  was 
to  use  multiple  regression  with  a  binary  response.  Fundamentally,  the 
approach  used  is  to  look  at  personnel  data  for  factors  that  describe  the 
difference  between  an  identified  set  of  suspected  experts  and  the  rest 
of  the  crew  force,  and  to  develop  a  probability  measure  that  any  given 
individual  is  a  member  of  the  expert  group.  It  must  be  kept  in  mind 
here  that  there  is  no  intent  to  ascribe  cause  and  effect  in  this  model. 
Using  the  model  to  allocate  more  flying  hours  (if  flying  hours  are  a 
factor  in  describing  experience)  to  individuals  with  low  experience 
scores  may  not  be  proper.  The  model  may  indicate  that  flying  hours  are 
useful  for  describing  uho  experts  are;  it  does  not  say  that  flying  hours 
cause  expertise. 

As  in  all  regression,  the  factors  to  be  used  in  this  particular 
formulation  of  a  multiple  regression  are  those  that  test  significant  in 
predicting  the  responses.  However,  Deming  suggests  that  traditional 
statistics  are  in  many  ways  poorly  suited  to  describing  non-static 
systems  (i.e. ,  systems  that  are  not  in  a  steady  state).  For  non-static 
systems,  he  suggests  that  confidence  intervals  should  not  be  an  over 
riding  consideration.  Rather,  emphasis  should  be  placed  on  detecting 
gross  patterns.  For  this  DSS,  with  a  goal  of  changing  the  system  it 
examines,  the  system  is  known  to  be  non-static.  Therefore,  significance 
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levels  for  determining  factors  impacting  experience  should  be  relaxed. 
The  goal  here  is  not  an  exact  answer,  but  rather  a  better  answer. 

552  AMACW/DO  supplied  a  partial  database  on  MCX^s  (with  only  a  few 
personnel  factors  that  could  possibly  bear  on  experience).  Most  of  the 
factors  available  in  the  database  were  related  to  time  in  some  manner; 
many  of  the  factors  discussed  in  the  data  section  that  should  help  to 
build  a  good  scoring  model  were  not  available. 

Using  this  data,  a  preliminary  examination  was  conducted  to  test 
the  usefulness  of  the  scoring  approach.  As  discussed  in  chapter  4, 
there  are  some  problems  with  the  technique  used  here  that  require 
remedial  measures.  One  of  the  problems  is  that  variance  is  non-uniform. 
This  was  addressed  by  using  a  weighted  regression. 

An  unweighted  regression  was  first  conducted;  the  resulting 
predicted  response  was  used  to  develop  the  appropriate  weignts 

w,  =  1  /  C  Y,<l-Y,)  1 

for  the  weighted  regression,  where  w,  is  the  weight  for  the  i*" 
observation  and  Y,  is  the  predicted  response  for  the  same  observation. 

Corrections  for  responses  out  of  the  allowed  range  were  not 
necessary  for  the  data  tested.  If  out  of  range  responses  had  been 
observed,  appropriate  transformations  are  available  to  convert  the 
formulation  to  a  probit  or  logit  regression.  If  this  were  not  desired, 
other  remedies  could  be  available,  depending  upon  the  factors,  and  their 
real  world  implications  for  experience.  As  discussed  previously, 
capping  the  maximum  and  minimum  values  certain  variables  can  assume  may 
make  sense. 

Results  for  the  regression  of  the  MCC  data  are  shown  below.  Tor 
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the  examination  conducted,  a  p  value  of  .4  was  accepted  as  significant. 
While  the  weighted  regression  only  accounts  for  30'/.  of  the  variance 
observed,  this  approach  was  still  viewed  as  feasible  (it  is  actually 
surprising  to  the  author  that  30*/.  of  the  variance  could  be  explained 
with  the  data  at  hand.  This  may  bode  well  for  the  confidences  that  can 
be  applied  with  better  data). 


FACTOR  LABELS 


D  =  YB£«S 

PRIOR  SERVICE 

DD  =  D* 

DI  =  Dxl 

interaction 

E  =  TII^ 

IN  SERVICE 

EE  =  E* 

X 

111 

11 

111 

interact  ion 

F  =  TIME 

IN  GRADE 

FF  =  F* 

IH  =  Hxl 

interaction 

H  =  VEflRS  ON  SHORT  TOURS 
I  =  YE/^  ON  LONG  TOURS 


This  is  the  regression  that  was  conducted  to  establ ish  weights. 
Factor  I  was  retained  (with  a  p  value  of  .61)  because  of  the  interaction 
that  were  significant.  R*  is  .21,  and  the  model  is  significant  at  .15. 


Table  2  -  Unweighted  Regression 


PREDICTOR 

VARIABLES 

COEFFICIENT 

STD  ERROR 

STUDENT'S  T 

P 

CONST  WT 

-4.6440 

1.5634 

-2.97 

O.OCVi? 

E 

6. 1335E-01 

1.9060E-01 

3.22 

0.0020 

F 

-2.4483E-01 

1.2412E-01 

-1.97 

0.0528 

I 

6.8520E-O2 

1.3411E-01 

0.51 

0.6111 

DD 

-2. 7636E-02 

l.Oei0E-O2 

-2.55 

0.0130 

EE 

-1.3576E-02 

4.6283E-03 

-2.93 

0.0046 

FF 

1,6030E-02 

9.5699E-03 

1.68 

0.0987 

DE 

1.Q232E-02 

1.3590E-O2 

1.34 

0. 1844 

DI 

2.2211E-02 

1.2307E-02 

1.80 

0.0757 

D 

-2.95696-01 

2.50816-01 

-1.14 

0.2574 

El 

-1.7528E-02 

1.0546E-02 

-1.66 

0. 1013 

H 

-3. 7225E-01 

2. 1815E-01 

-1.71 

0.0927 

HI 

1.3263E-01 

6.3068E-02 

2. 10 

0.0393 

OVERALL  F 
R  SQUARED 


1.408 

0.2155 


P  VALUE 


0. 1514 


As  can  be  seen  here,  there  is  not  a  large  separation  in  the 
[experience  scores  of  the  two  populations.  The  lower  grouping  is  for 
the  unknown  portion  of  the  crew,  v«4nile  the  upper  grouping  is  for  the 
instructors  and  StanEval  personnel.  While  the  upper  group  has  more 
occurrences  (as  a  percent  of  the  upper  group)  at  the  high  end  of  the  X 
axis,  the  range  extend  all  the  way  to  0  (approximately). 


The  second  regression,  using  the  weights  developed  in  the  first, 
has  an  elevated  R*  (*  .30)  and  a  p  value  of  .013. 
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Table  3  -  Weighted  Regression 


PREDICTOR 

VARIABLES 

COEFFICIENT 

STD  ERROR 

STUDENT'S  T 

P 

CONSTANT 

-5.2769 

1.3196 

^.00 

0.0002 

E 

6.7984E-01 

1.6138E-01 

4.21 

0.0001 

F 

-2. 2977E-01 

1.0795E-O1 

-2.13 

0.0371 

I 

1. 1782E-01 

1. 1150E-01 

1.06 

0.2946 

DD 

-2.4499E-02 

1.0397E-02 

-2.36 

0.0215 

EE 

-1.5422E-02 

3.9321E-03 

-3.92 

0.0002 

FF 

1.5526E-02 
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As  can  be  seen  in  Figure  28,  the  result  of  weighting  the  regression 
was  to  better  separated  the  two  groups.  The  center  of  mass  of  the  upper 
group  has  shifted  higher  on  the  X  axis,  uhile  that  of  the  lower  group 
has  moved  down  the  axis. 


Capabilities  Models 

The  capabilities  model  falls  into  the  realm  of  MCDM,  since  it  deals 
with  balancing  multiple  objectives.  This  model  should  only  address 
capabilities  as  impacted  by  personnel  issues. 

AWACS  capabilities  as  a  function  of  personnel  is  driven  by  multiple 
personnel  issues,  such  as;  1)  are  there  enough  people  to  man  a  crew 
position;  2)  are  there  enough  experienced  people  in  a  crew  position 
(average  issues);  3)  are  there  enough  experts  to  handle  exceptional 
cases  and  to  pass  on  knowledge  (distribution  issues);  4)  are  there 
particular  crew  positions  that  are  critical  in  impacting  capabilities. 
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and  5)  i f  so,  do  they  need  special  attention  to  fix  their  deficiencies 
relative  to  the  rest  of  the  crew  positions  (allocation  of  scarce 
resources) . 

The  objective  of  this  model  is  not  to  provide  an  absolute  answer  on 
capability.  Rather,  the  goal  is  to  specify  some  measure  that  allows 
trends  to  be  assessed.  As  such,  the  scale  for  "CAPABILITY"  is  not 
critical,  but  is  sure  to  be  a  management  stumbling  block  if  some 
reference  is  not  provided. 

Approaches  for  providing  reference  include: 

1)  defining  the  100  level  to  be  1007.  manning; 

2)  with  a  specified  experience  distribution  (use  the  same 
distribution  against  uhich  experience  profiles  are  measured  by 
acquisition  managers),  and; 

3)  with  a  "table"  of  trade  offs  between  changes  in  these  values 


142 


clearly  stated.  Eliciting  these  trade  offs  (utilities  and  weights)  is 
critical,  and  may  well  be  the  most  difficult  portion  in  developing  this 
model.  Every  one  of  the  assumptions  made  in  estimating  trade  offs 
between  these  factors  must  be  available  to  the  decision  makers  that  use 
the  model . 

The  discussion  above  presents  general  guidelines  for  model 
development.  Developing  them  will  be  a  difficult  and  time  consuming 
task. 
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APPEhDIX  F 


MZOn  Examination 

This  appendix  discusses  the  examination  made  of  MCDM  techniques  for 
the  purpose  of  assessing  experience  levels  in  the  different  crew 
positions.  While  these  methods  were  not  adopted,  they  are  applicable  to 
other  portions  of  the  overall  DSS.  For  example,  MADA  methods  will 
probably  be  necessary  if  a  reference  (desired)  experience  profile  is  to 
be  developed.  Also,  MCO  techniques  (not  discussed  in  this  appendix, 
since  the  area  of  concern  was  of  a  descriptive  rather  than  prescriptive 
nature)  will  surely  be  necessary  for  developing  the  capabilities 
assessment  models. 
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Larger  System  Isanp^s 

The  overall  problem  addressed  in  this  research  is  the  development 
of  an  effective  decision  aide  to  support  AUJACS  air  crew  managers  in 
determining  personnel  acquisition  needs.  (The  manager’s  goal  is  to 
maintain  AUIACS  with  an  crew  force  that  can  effectively  execute  the  AWACS 
mission) . 

The  acquisition  requirements  of  AWACS  are  driven  by  two  issues:  1) 
the  number  of  people  required  to  man  a  crew  position,  and;  2)  the 
experience  these  people  must  possess  for  an  AUIACS  mission  to  be 
effective.  The  first  issue  is  concerned  with  personnel  flow  through  the 
crew  system.  Managers  attempt  to  keep  crew  positions  fully  manned. 
GLiantitative  shortages  due  to  mis-timing  of  acquisitions  (to  fill 
manning  vacancies)  need  to  be  minimized.  The  second  issue  is  concerned 
with  the  distribution  of  experience  in  the  crew  force.  All  members  of  a 
crew  position  need  not  be  expert,  but  some  minimum  set  must  be.  There 
is  also  some  limit  to  the  nuniaer  of  novices  the  position  can  tolerate 
before  the  AUIACS  mission  becomes  ineffective  due  to  inexperience  in  the 
crew  position. 

The  overall  measure  of  a  successful  manning  acquisition  program  is 
how  capable  AUIACS  is  of  performing  its  mission  (limited  to  effects  based 
on  personnel  considerations).  The  AUIACS  capabilities  issue  involves 
some  balance  among  the  above  mentioned  factors  (absolute  numbers  and 
experience).  If  the  air  crew  managers  are  to  be  successful  in  managing 
acquisition,  they  must  be  able  to  balance  experience  and  number  issues 
to  arrive  at  the  best  acquisition  strategy. 

In  the  overall  problem,  "optimizing"  AUIACS  capabilities  is  a  Multi- 
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Criteria  Optimizaticxi  (MCO)  problem,  possibly  addressed  best  as  a 
compromise  program.  There  are  two  criteria  for  each  crew  position  in 
this  problem.  These  criteria  are; 

1)  The  absolute  manning  level.  The  "ideal"  and  "goal"  values  for 
this  criterion  are  the  same.  It  is  the  manning  authorization  level  for 
the  crew  position,  as  specified  by  higher  headquarters. 

2)  The  distribution  of  the  experience  in  the  crew  position.  The 
mean  experience  level  of  the  personnel  in  the  crew  position  is  a 
possible  measure  of  composite  experience  in  the  crew  force.  It  can 
provide  a  basic  measure  of  the  ability  of  the  current  crew  force  to 
accomplish  the  AWACS  mission  effectively.  The  "goal"  for  the  mean 
experience  level  will  need  to  be  extracted  from  the  AUJACS  managers,  and 
can  be  referenced  against  a  constructed  experience  scale  ranging  from  0 
to  1,  Unere  a  .9  or  greater  is  the  "experience  score"  of  the  "ideal 
expert".  (The  distribution  of  experience  is  also  critical,  particularly 
with  respect  to  a  heavy  representation  in  the  crew  force  of  novices,  or 
a  light  representation  of  "expert"  experts.  However,  all  distribution 
issues  will  not  be  considered  initially.  Refinements  of  the  models  in 
the  DSS  will  be  necessary  to  account  for  more  than  central  tendency. 

Some  other  moment  will  undoubtedly  need  to  be  considered  to  ensure 
unacceptable  experience  distributions  in  the  crew  force  do  not  occur. 

The  introduction  of  a  "distribution"  criteria  in  no  way  changes  the 
basic  approach  needed;  it  only  causes  the  dimensionality  of  the  MCO 
problem  to  increase). 

For  managers  to  structure  personnel  acquisitions  in  a  manner  that 
"optimally"  balances  (under  some  MCO  methodology)  the  two  criteria  just 
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discussed,  models  are  required  that  show  the  impact  of  a  manning 
acquisition  decision  on  each  of  the  criteria.  A  first  step  is 
identifying  the  factors  that  bear  on  each  of  the  criteria,  i.e. , 
identifying  the  attributes  in  the  alternative  space  of  each. 

The  Kamel  Problem 

The  kernel  of  the  overall  problem  (discussed  above)  concerns  the 
"experience"  issue.  For  most  of  the  crew  positions,  there  are  no  good 
measures  of  the  crew  member’s  competence  in  his  job  performance.  The 
goal  in  the  kernel  development  is  to  find  and  combine  some  set  of 
factors  in  an  "experience  score"  that  will  provide  a  surrogate  measure 
of  the  individual  crew  member’s  effectiveness  at  his  job.  (The 
individuals  to  be  "scored"  are  the  AUIACS  air  crew  members  within  each 
crew  position.  Ratings  are  only  relevant  within  the  crew  position,  not 
between  positions.  The  score  of  interest  is  a  rough  estimate  of  each 
individual’s  ability  to  competently  handle  all  of  the  myriad  of 
contingencies  that  face  crew  members  in  his  position  in  an  AkIACS 
mission).  To  establish  "experience  scores",  three  tasks  must  be 
accomplished.  These  tasks  are  to: 

1)  establish  a  set  of  factors  that  are  reasonable  indicators  of 
performance.  Examples  of  possible  candidate  factors  are:  1)  rank,  2) 
years  of  service,  3)  Standardization  Evaluation  (StanEval)  ratings,  4) 
number  of  flying  hours,  5)  number  of  missions  flown,  6)  number  of 
exercise  missions  flown,  etc.; 

2)  determine  appropriate  weights  among  the  factors  (if  the  experts 
feel  all  are  not  of  the  same  weight),  and; 

3)  determine  utility  functions  for  each  factor  that  describe  the 
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relationship  between  the  states  of  the  factor. 

The  "experience  score"  that  is  generated  for  each  crew  meniber  will 
become  part  of  his  database  attributes,  so  that  the  experience  level  of 
the  cvew  position  as  a  whole  can  be  judged  by  simple  graphic 
representations  of  the  database  contents.  After  a  valid  scoring  method 
is  arrived  at,  the  experience  profile  of  the  crew  force  can  be  presented 
to  the  manager  to  enhance  his  acquisition  recommendations.  He  will  be 
able  to  identify  uhat  impact  policies  controlling  acquisitions  and 
losses  have  on  the  experience  profile  of  the  AUIACS  crew  position.  He 
will  be  able  to  assess  issues  such  as: 

1)  whether  he  needs  to  take  special  actions  to  recapture  experience 
(by  identifying  past  Ali^lACS  members  with  the  requisite  experience  who  are 
at  large  in  the  AF  manning  pool,  and  whom  MPC  can  reassign  to  AWACS  to 
correct  an  imbalance),  or  whether  he  can  "grow"  that  experience; 

2)  whether  he  can  live  with  a  manning  acquisition  that  is  composed 
entirely  of  novices,  or; 

3)  whether  he  must  get  MPC  and  TAC  to  give  him  some  mix  of 
personnel  that  balances  novice  and  experienced  "acquisitions". 

m  Assessment  of  a  MUlti-Criteria  Decision  Analysis  (MADA)  ftxiroach  to 
the  Kernel 

liADA,  in  this  context,  is  oriented  to  the  determination  of 
experience  based  on  expert  opinion.  Here,  the  objectives  are  to  elicit 
appropriate  experience  factors  from  "the  experts"  and  to  determine  the 
weights  and  utilities  the  experts  hold  for  the  factors.  In  general,  the 
possible  factors  are  only  indicators  of  experience;  rigorous  descriptors 
of  experience  are  not  available.  With  no  rigorous  measures  of 
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experience,  it  is  critical  that  a  good  set  of  indicators  be  identified, 


and  their  relative  importance  assessed. 

The  experts  to  be  consulted  are  managers  of  the  personnel  to  be 
rated.  CFor  the  purposes  of  assessing  MADA  in  this  research,  the 
"experts"  consulted  were  two  Weapons  Directors  (WD)  and  two  former  AWACS 
analysts  were  available  locally.  The  crew  position  considered  was 
the  WD  position!.  Initially,  the  set  of  possible  factors  must  be 
limited  to  those  for  which  data  is  collected  as  an  attribute  of  each 
individual.  (The  specification  of  a  factor  from  the  set  of  all 
conceivable  factors  is  not  useful  at  this  point  in  time.  If  an  expert 
feels  that  some  attribute  is  valuable  in  determining  experience,  and 
data  is  not  currently  collected  for  the  desired  attribute,  the 
identification  can  be  usf?d  to  instigate  data  collection.  Better  factors 
can  be  incorporated  into  the  model  at  some  later  time). 

The  formulation  of  "experience  score"  needed  in  the  kernel  and  the 
MADA  methodology  for  arriving  at  the  required  weights  and  factors  are 
presented  in  the  following  sections. 

Formulation  for  an  "Experience  Score" 

The  formulation  of  the  "experience  score"  to  be  developed  for  each 
crew  member  is  a  simple  additive  model: 

E  =  Z  C  <w'),  ]  C  u,(aj)  ]  , 

I 

where  E  is  the  "score",  w/  is  the  "standardized"  weight  for  the  i**" 
factor,  and  Uj(a)  is  the  i*"  "utility  value"  that  is  a  function  of  the 
i,^  attribute  level  in  an  individual  crew  member’s  attribute  vector.  In 
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the  above  formulation,  and  are  the  weights  and  utilities  that  must 
be  extracted  from  the  experts.  (Here,  w/  is  w,  "standardized"  to  give 
a  maximum  value  of  1  to  the  score  E  wtien  UjCaj)  =  1  for  i  = 
where  q  is  the  number  of  factors  in  the  model;  in  other  words,  the 
standardized  weight  is  w,*=  C  w,  D  /  C  Ew,  D  ).  This  model,  Uiile  only 
being  applied  within  a  crew  position,  can  be  written  in  a  general  form 
that  will  score  all  crew  positions.  It  does  not  have  to  be  specialized 
by  crew  position.  It  can  be  written  incorporating  all  factors 
identified  for  each  crew  position.  All  that  is  required  to  get 
consistent  scores  in  a  specific  crew  position  is  to  maintain  unique 
weighting  and  utility  function  vectors,  and  set  the  weight  of  any  factor 
that  is  not  appl icable  to  O,  v^ich  removes  the  factor  from  the  scoring 
model . 

Eliciting  Factors  Bearing  on  Experience 

To  elicit  a  set  of  possible  factors  from  the  experts,  surveys  were 
conducted.  The  surveys  had  two  portions,  one  based  on  Semantic  Scales 
and  the  other  a  Pair  (Comparison  of  factors  CChanl. 

Part  One  of  the  Survey.  Part  one  of  the  survey  contains  a  set  of 
possible  factors  that  may  be  of  value  and  for  uhich  data  is  known  to 
exist.  The  survey  respondents  completed  two  different  evaluations  of 
the  factors. 

The  first  evaluation  of  the  factors  was  a  semantic  scaling  survey 
to  elicited  perceived  importance  rankings  for  the  factors.  Rankings  are 
on  a  five  point  scale  (seven  point  scales  are  often  used)  where  the 
extremes  of  the  scale  are  described  verbally.  To  complete  the  survey, 
the  respondents  were  asked  to  establish  rankings  independently  of  all 
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other  factors. 


The  second  evaluation  of  the  factors  was  based  on  pair-wise 
comparisons  of  the  factors.  All  factors  are  paired,  and  the  respondents 
required  to  choose  the  preferred  factor  from  the  pair. 

The  two  portions  of  this  section  of  the  survey  have  two  purposes. 
These  are:  1)  to  assess  the  importance  of  the  various  factors  (both 
portions),  and;  2)  to  check  for  consistency  in  the  rankings.  This  is 
accomplished  by  comparing  the  two  portions  of  the  survey,  and  by 
assessing  the  transitivity  of  the  factor  rankings  in  the  pair-wise 
comparison. 

In  the  scheme  of  the  overall  survey,  the  first  portion  serves  two 
further  purposes.  First,  it  may  produce  a  good  set  of  factors  on  its 
own,  and  second,  it  will  get  the  expert  thinking  on  the  experience 
factor  selection  issue. 

Part  Two  of  the  SLon/ey.  The  second  portion  of  the  survey  is 
basically  unstructured.  It  consists  of  an  open  ended  request  for 
factors  the  experts  deem  important,  such  that  the  factors  suggested  are 
at  least  as  valuable  to  the  respondent  as  the  most  important  factor 
identified  in  part  one.  This  will  mainly  be  used  as  a  means  of 
identifying  possible  new  factors  that  have  not  been  considered  in  the 
first  section  of  the  survey.  However,  under  certain  conditions,  these 
factors  may  be  included  in  the  model  to  be  developed. 

Selecticn  of  Factors.  The  selection  of  factors  will  be  based  on 
two  criteria.  For  factors  evaluated  in  the  first  portion  of  the  survey, 
any  factor  that  is  in  the  top  two  categories  of  the  majority  of  the 
responses  will  be  included.  This  selection  criterion  is  used  for  the 
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following  reasons:  1)  the  response  sanple  will  be  to  small  for 
statistical  methods  to  be  useful,  and;  2)  if  a  majority  of  the 
respondents  agree  that  the  factor  is  of  above  average  importance,  it 
should  be  considered  until  a  larger  response  can  be  achies/ed. 

For  the  factors  identified  in  the  second  portion  of  the  survey,  any 
factor  that  appears  on  two  or  more  surveys  will  be  included.  Here,  the 
selection  criterion  is  more  liberal  than  in  the  first  portion  of  the 
survey.  This  is  done  because  if  some  factor  appears  on  more  than  one 
survey,  solely  from  the  respondents  assessment  of  uhat  is  important  and 
independent  of  survey  prompts,  the  factor  is  probably  significant. 

(While  an  explicit  pair-wise  comparison  is  not  conducted,  guidance  will 
be  given  that  steers  the  respondent  toward  a  mental  pair-wise  assessment 
of  the  relative  strengths  of  the  factors.  This  being  so,  a  duplicate 
identification  of  the  same  factor  will  be  considered  sufficient 
Justification  for  selecting  the  factor.  Given  more  time  and  resources, 
this  initial  effort  would  be  used  as  a  pilot  survey,  and  further 
surveying,  incorporating  the  results  of  the  pilot  and  with  a  larger 
sample,  would  be  conducted.  For  the  purposes  of  assessing  MADA  as  an 
approach,  this  was  not  done). 

Eliciting  Factor  Vteiohts  and  Utilities 

Appropriate  relative  weights  for  the  factors,  and  the  utility 
function  that  describes  the  relationship  of  the  states  within  each 
factor,  can  be  determined  through  a  series  of  reference  lotteries. 

(This  was  discussed  in  the  methodology  portion  of  the  main  text,  but  is 
discussed  here  again). 

In  a  "basic"  reference  gamble  procedure,  sequences  of  reference 
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ganibles  are  conducted  u^ere,  prior  to  the  gamble,  payoffs  for  the 
reference  gamble  are  established.  Then,  iteratively,  probabilities  (p) 
for  winning  the  gamble  are  assigned,  and  the  certainty  equivalent  (CE) 
for  the  gamble  evaluated.  Plotting  p  verses  CE  yields  the  utility  curve 
for  the  factor  being  examined  CHolloway  :  4223.  (Holloway  also 
describes  a  variation  on  the  basic  reference  gamble  that  fixes  a 
sequence  of  certain  values  (CV)  and  then  determines  the  probability  p  at 
which  the  decision  maker  is  indifferent  to  the  lottery.  Plotting  p 
against  CV  again  yields  the  utility  curve). 

The  reference  gamble  to  be  used  is  a  50-50  gamble.  This  is  another 
modification  of  the  basic  reference  gamble.  (Holloway  suggests  that 
this  is  often  a  better  gamble,  since  people  often  thing  in  terms  of  go- 
no  go  choices  C4203).  Here,  GEs  are  first  determined  for  p  =  0,  .5,  and 
1.  For  the  rest  of  the  procedure,  p  is  fixed  at  .5  and  the  (3£s  for  each 
gamble  elicited.  The  probability  for  the  certain  value  can  be 
determined  from  the  gamble  conducted.  The  50-50  gamble  was  used  to 
determine  both  the  factor  weights  and  the  utility  functions  that 
describe  the  factors.  This  was  done  in  an  automated  question  and  answer 
session.  A  sequence  of  structured  questions  were  asked  of  the  experts, 
and  appropriate  lotteries  conducted.  The  results  of  the  sessions  were  a 
set  of  weights  and  utility  functions  to  be  used  in  the  additive 
"Experience  Score"  described  first  in  the  "Formulation"  section  of  this 
f4]pend  i  x . 

The  software  package  "SmartEdge"  (ver  3.0)  was  used  to  accomplish 
the  structured  question  and  answer  sessions.  Prior  to  the  experts  being 
asked  to  participate  in  the  SmartEdge  session,  initial  setup  work  was 
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ccampleted  on  a  "Decision  File"  so  that  nothing  was  required  of  the 
experts  but  that  they  answer  a  the  programs  sequence  of  questions  (i.e. , 
all  entries  required  up  to  the  "DECISION  MAKER  DATA"  were  entered). 

In  a  SmartEdge  session,  the  expert  being  interviewed  answered 
questions  in  the  "DECISION  MAKER  DATA"  segments  of  the  software  package. 
The  results  of  the  interview  session  defined  the  expert's  utility 
function  for  each  attribute  listed.  (A  similar  interview  develops  the 
appropriate  weights  for  the  factors). 

Survey 

The  survey  included  in  at  the  end  of  this  appendix  (Tab  A)  is  aimed 
at  eliciting  a  set  of  factors  that  indicate  experience  in  WDs.  It  was 
completed  by  four  "experts".  The  results  of  the  survey  are  presented 
next. 

Survey  Results  for  Part  I<a).  The  results  of  the  survey  for  Part 
(a)  shown  in  Table  D-1. 


Table  4  -  Survey  Results,  Part  1(a) 


FACTOR  (REEPGMlENr  » 

1 

IMWTANCE 

2  3 

4 

(TOTAL) 

A) 

StanEval  ratings 

4 

5 

5 

5 

(19) 

B) 

E-3  flying  hours 

2 

3 

4 

4 

(13) 

C) 

Years  in  position 

4 

3 

4 

1 

(12) 

D) 

Instructor  qualified 

4 

5 

5 

5 

(19) 

E) 

#  of  E-3  missions 

3 

3 

4 

3 

(13) 

F) 

#  of  exercise  missions 

4 

4 

4 

4 

(16) 

G) 

#  of  Simulator  sessions 

2 

2 

3 

3 

(10) 

H) 

Training  evaluations  ... 

3 

4 

3 

3 

(13) 

I) 

...  ground  controller  ... 

4 

4 

2 

3 

(13) 

Based  on  the  selection  criteria  stated  previously  in  this  appendix, 
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factors  A,  D  and  F  appear  to  be  a  good  set  of  indicators  for  WD 
experience.  As  stated  previously,  with  a  larger  sample,  a  statistical 
evaluation  of  the  results  would  be  valuable.  From  the  above  table,  it 
looks  like  factors  E  ,  H  and  I  may  also  be  possible  experience 
indicators.  A  larger  sample  and  a  statistical  measure  would  help 
determine  if  they  are  indeed  useful  indicators. 

Survey  Results  for  Part  Kb).  Evaluation  of  the  results  of  Part 
1(b)  resulted  in  a  change  in  the  factor  selection  criteria,  and  in  the 
factors  selected.  Examination  of  the  paired  comparisons  stiowed  that 
three  of  the  respondents  surveyed  considered  the  set  of  factors  to  be 
fully  transitive.  Further,  the  preference  structure  established  by  the 
comparisons  were  consistent  with  each  respondents  answers  in  Part  1(a) 
of  the  survey  Cwithin  the  resolution  of  the  scale  of  Part  I (a) 3.  (The 
transitivity  of  the  factors  was  established  on  a  spreadsheet.  One 
compound  sort  of  the  factors  ranked  the  respondents  preference  for  the 
factors.  The  sorted  responses  for  one  survey  are  included  in  Tab  B). 

The  fourth  respondent's  survey  results  demonstrated 
intransitivities  and  also  indicated  inconsistencies  with  his  responses 
in  Part  1(a)  of  the  survey.  This  respondent  was  not  available  for 
further  questioning  (member  was  TOY)  that  might  have  revealed  uhy  the 
intransitivities  existed  and  why  there  was  not  good  correlation  pattern 
between  his  Part  1(a)  and  Part  Kb)  responses.  His  responses  were 
therefore  removed  from  considerat ion  in  the  study. 

Reduced  Response  Set.  While  three  of  the  respondents  were 
internally  consistent  n  their  rankings,  their  rankings  in  the  pair-wise 
comparison  did  not  agree  fully  with  each  other.  Two  agreed  exactly  in 


156 


the  ranking  of  the  top  four  factors.  The  third  included  only  one  of 
these  factors  (A)  in  his  first  four  choices.  The  factor  preferences 
established  by  the  pair-wise  comparison  are; 

1)  A>I>D>C>F>E>H>B>G 

2)  D>A>F>E>C>I>B>H>G 

3)  D>A>F>E>H>I>C>B>G 

Survey  Clarification.  Further  questioning  of  the  respondents 
showed  that  there  was  a  fundamentally  different  perception  about  the 
scenario  under  which  they  were  operating  in  Part  Kb)  of  the  survey. 

The  two  respondents  who  were  most  in  agreement  operated  from  the 
assumption  that  there  was  no  guarantee  that  any  of  the  crew  members  they 
were  to  choose  from  would  have  ground  control  experience,  and  would 
therefore  not  attempt  to  base  a  decision  on  this  factor.  Further,  they 
both  indicated  that  this  assumption  was  based  on  their  knowledge  that 
there  are  not  many  people  in  the  AWACS  community  with  ground  control 
experience.  (When  asked  if  this  wasn’t  also  true  for  basing  a  decision 
on  instructor  qualification,  they  indicated  that  they  were  indeed 
operating  from  the  same  assumption.  However,  the  indicated  that  1) 
there  are  many  more  people  in  AWACS  v>io  have  been  instructors  at  some 
time,  so  they  felt  there  was  a  much  better  chance  that  pool  of  crew 
members  to  be  drawn  from  would  include  members  with  instructor 
backgrounds,  and  2)  that  instructor  qualification  was  such  a  good 
discriminator  of  expertise,  that  they  were  willing  to  "gamble" 
on  finding  someone  with  this  background). 

The  respondent  who  was  not  in  agreement  with  the  other  two,  on 
considering  these  issues,  indicated  that  factor  I  would  move  down  in  his 
rankings  if  it  were  unknown  if  any  crew  members  in  the  pool  to  be  drav»r\ 
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from  had  ever  been  ground  controllers.  He  had  based  his  rankings  on  the 
assumpt ion  that  the  pool  to  be  drav^  from  contained  members  in  every 
factor  category. 

Survey  Results  of  Part  II.  Part  II  of  the  survey  produced  two  more 
possible  factors.  One  factor  appeared  on  three  surveys  and  one  appeared 
»  on  two.  These  new  factors  were: 

1)  Has  the  WD  ever  been  a  StanEval  flight  performance  evaluator. 

If  so,  this  is  a  highly  reliable  indicator  of  experience,  based  on 
comments  included  on  the  surveys.  This  factor  appeared  on  three 
surveys. 

2)  Has  the  member  attended  the  Fighter  Weapons  School  (FWS)  or  the 
Tactical  Air  Traffic  Control  (TACATC)  training  courses.  If  so,  this  is 
also  a  reliable  indicator  of  experience.  This  are  very  limited  courses 
in  advanced  control  tactics  that  only  a  chosen  few  get  into.  Special 
schools  appeared  on  two  surveys. 

Independence  Issues.  During  the  survey  clarification  discussed 
above,  another  issue  that  came  out  had  to  do  with  the  independence  of 
the  factors.  Consideration  of  the  factor  set  led  to  the  following 
conclusions; 

1)  number  of  missions  flown,  number  of  exercise  missions  flown, 
and  number  of  simulator  sessions  could  be  considered  to  be  largely 
independent,  but  all  three  were  related  to  number  of  years  in  the  crew 
posit  ion; 

2)  factors  like  instructor  qualification,  special  school 
attendance  and  the  respondent  suggested  standardization  evaluator 
qualification  factor  were  not  independent  of  most  of  the  other  factors, 
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and; 

3)  any  crew  members  ubo  were  instructor  or  standardization 
qualified  or  vAio  had  attended  any  of  the  special  schools  could  be 
considered  experts  because  of  the  screening  they  had  been  through  before 
they  were  selected  for  these  positions.  Members  selected  for  these 
functions  were  highly  qualified  before  their  selection;  the  experience 
gained  by  filling  the  positions  or  going  to  schools  only  reinforced 
their  expertise. 

Fundamentally,  member’s  vtio  were  ever  instructors  or 
standardization  evaluators,  or  ^>io  had  been  to  special  schools  should  be 
considered  "experts",  and  removed  to  a  separate  "pot".  These  factors 
can  be  used  in  a  Lexicographic  screening  of  the  AO  population.  They  are 
sure  indicators  that  a  member  is  “experienced".  The  Instructor 
qual i fication  and  Standardization  Evaluator  qualification  factors  can  be 
rolled  into  one  factor  that  guarantees  preference  for  AOs  with  either 
qualification,  and  the  Special  Schools  factor  can  be  used  as  a  second 
factor  in  the  lexicographic  ordering  of  “experts"  with  attendance  of 
more  special  schools  preferred.  (Further  distinction  among  experts  may 
be  possible.  The  survey  participants  felt  the  only  factors  among  those 
listed  or  elicited  that  might  distinguish  experience  among  the  experts 
would  be  B,  E,  and  possibly  F>. 

Altered  Factor  Selection.  Given  the  above  discussion,  the  criteria 
for  factor  selection  was  altered  to  pick  the  factors  that  met  the 
original  criteria  and  that  had  minimum  score  of  3.  When  there  were  four 
respondents,  and  all  three  of  the  four  agreed  that  a  factor  was  of  above 
average  importance,  the  presence  of  an  "outlier"  could  be  tolerated. 
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With  only  three  respondents  left  in  the  sample,  the  criteria  was  altered 
so  that  one  rating  of  below  average  for  a  factor  raises  sufficient 
question  about  its  worth  to  remove  it  from  consideration.  Further, 
having  identified  that  factor  C  was  not  independent  of  some  of  the  other 
measures,  it  was  removed  from  the  set  of  factors  to  be  considered. 
Finally,  the  rating  question  was  divided  into  two  areas:  1)  separate 
out  the  guaranteed  "experts"  using  factor  D  (and  experience  as  a 
Standardization  Evaluator  and  attendance  of  one  or  more  of  the  premier 
controller  schools,  as  discussed  in  the  next  section). 

The  results  of  Part  1(a)  of  the  survey  with  the  inconsistent 
respondees  data  removed  are  shown  in  Table  D-2,  below.  The  factors  that 
meet  the  selection  criteria  are  indicated. 


Table  3  -  Survey  Results,  Part  1(a),  Modified 


IfPORTflNCE 

FAC7CR  (REEPOVDENT  »1  2  3  (TD 

StanEval  45  5 

Fly  Hours  2  3  4 


(TDTPL) 

NOW 

(14) 

* 

(  9) 

INCLUDED 

NOW  PREVIOUSLY 
*  * 


. 3]3:s:s 


The  pair-wise  comparison  results  with  factors  C,  D  and  I  removed  (C 
for  non- independence,  D  because  it  will  be  used  as  part  of  a 
Lexicographic  screening  of  the  WD  population,  and  I  because  of  the 
difference  in  respondent  treatment  of  the  factor)  is  presented  below. 
Transitivity  of  the  results  for  each  respondent  is  maintained  since 
removal  of  factors  causes  no  changes,  and  no  new  factors  were  added. 
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1)  A>F>E>H>B>G 

2)  A>F>E>B>H>G 

3)  A>F>E>H>B>G 

Using  the  above  information,  SmartEdge  was  configured  to  elicit  the 
weights  and  utilities  of  the  remaining  factors.  The  SmartEdge  sessions 
are  discussed  next. 

SmartEdge  Session 

Assessment  of  utilities  and  weights  for  use  in  a  value  function 
formulation  was  accomplished  in  a  SmartEdge  session  with  the  two 
remaining  factors,  i.e. ,  A)  StanEval  ratings,  and  F)  #  of  exercise 
missions.  (This  assessment  is  done  for  the  WD  population  left  after  the 

population  has  been  screened  for  "experts"  as  discussed  above). 

Independence  issues  are  treated  at  the  Utility  Independence  level 
by  SmartEdge.  Preferential  Independence  is  not  explicitly  checked, 
since  given  Utility  Independence  "it  is  difficult  to  construct  a 
realistic  example  v^ere  pair — wise  utility  independence  would  be 
violated"  CSmartEdge  Documentation  :  B-lll.  Given  that  the  set  of 
factors  has  been  reduced  to  two,  this  is  a  moot  point.  (With  more  than 
two  factors.  Preferential  Independence  may  have  to  be  verified  prior  to 
using  a  package  like  SmartEdge).  Utility  Independence  and  Preferential 
Independence  are  the  same  in  this  case. 

The  SmartEdge  session  was  conducted  by  defining  a  set  of 
alternatives  in  the  CASE  DEFINITION  area  of  the  program.  The 
alternatives  in  this  case  are  "WDs"  to  be  ranked  by  preference. 
("Representative  WDs"  were  built  at  10  different  levels  of  experience. 
Each  was  assigned  representative  attribute  values  that  might  be  expected 
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at  these  levels  of  experience).  This  constituted  the  necessary  set-up 
work  to  allow  the  session  to  enter  the  weight  and  utility  determination 
phase. 

Establishing  utilities  for  each  factor  and  the  weights  between 
factors  entailed  the  surrogate  decision  maker  (only  one  of  the  WDs  had 
time  for  the  SmartEdge  session)  passing  through  two  sequences  of 
reference  lotteries.  Utilities  were  arrived  at  by  establishing  the 
decision  maker's  (DM)  trade-offs  within  a  specific  factor,  and  the 
weights  were  developed  by  assessing  preferences  expressed  by  gambles 
between  specified  trade-offs  between  the  factors. 

hlaving  produced  the  utilities  and  weights,  SmartEdge  was  asked  to 
rank  the  representative  WDs  that  were  defined  as  the  alternative  space 
of  the  problem.  All  were  ranked  in  the  order  expected,  and  the  assigned 
utility  scores  were  proportional  to  the  expected  difference  in 
experience  levels  among  the  alternatives.  Data  for  real  WDs,  obtained 
from  Tinker  AFB,  OK,  was  then  entered  into  the  alternative  space,  and 
the  program  was  executed  again.  The  real  WDs  were  also  ranked 
appropriately  (Tab  C) . 

Conclusions 

The  methodology  examined  here  gave  results  that  made  sense. 

However,  there  are  probably  many  better  discriminators  of  experience  for 
the  "non-experts"  that  should  be  examined.  As  these  factors  are 
included,  the  dimensionality  of  the  problem  increases  rapidly,  making 
solutions  more  difficult.  Further,  uhen  multiple  experts  differ  on 
utilities  and  weights,  MADA  has  some  problems.  Various  concordance 
measures  are  available  for  determining  whether  differences  observed 
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between  the  elicited  values  are  significant.  Uh fortunately,  Jhen  the 
concordance  measures  indicate  significant  disagreement,  MADA  currently 
has  no  generally  accepted  means  for  resolving  the  dispute. 

Possibly  a  more  significant  problem  in  this  instance  is  the  level 
of  expert  involvement  required  to  implemerit  this  type  of  strategy,  and 
the  difficulty  the  experts  have  in  understanding  the  utility  and 
weighting  elicitation  process.  For  this  simple  test,  weights  and 
utilities  were  only  developed  for  two  factors.  This  process  took  over 
six  hours  to  accomplish,  using  a  reasonably  friendly  software  package. 
The  major  problem  encountered  was  that,  even  using  a  50-50  gamble  (which 
Holloway  indicated  is  generally  the  most  easily  understood),  the 
decision  maker  being  queried  could  not  build  a  utility  curve  that 
matched  his  estimations,  even  t^en  he  had  a  clear  perception  of  uhat  the 
curve  should  look  like.  People  in  general  do  not  deal  with 
probabilities  well,  and  this  experience  indicates  they  have  particular 
problems  estimating  the  relative  probabilities  for  two  different  bounded 
ranges  (not  including  either  extreme)  in  the  interior  of  the  space  they 
are  considering. 

The  time  issue  is  critical  for  the  DBS.  The  people  who  will  be 
queried  in  this  process  at  Tinker  AFB  are  1)  non-technical ,  and  2) 
flyers  who  have  a  erratic,  and  very  full,  schedules.  Attempting  to 
corral  the  personnel  support  from  the  experts  to  repeat  the  MADA 
approach  for  12  crew  position  with  multiple  factors,  multiple  times  as 
the  system  evolves,  will  probably  kill  the  DBS  development  outright.  As 
such,  this  approach  should  only  be  used  as  a  last  resort,  when  other 
less  time  intensive  (flyer's  time)  methods  have  been  found  insufficient. 
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Having  said  this,  it  must  also  be  said  that,  <(iiile  these  techniques 
are  difficult  to  apply  to  the  current  problem,  they  do  offer  an 
alterative  means  of  looking  at  the  personnel  data  that  applies  to  the 
problem.  The  process  of  dealing  with  experts  is  illuminating,  and 
results  in  a  much  better  understanding  of  a  problem  than  results  from  a 
bare  examination  of  personnel  data  using  statistical  techniques.  As 
such,  there  is  some  value  in  an  analyst  "squandering"  (from  the  expert’s 
perspective)  a  little  expert  time  to  gain  the  insights  can  be  gleaned 
using  these  techniques. 


1&4 


Tab  A  —  The  Survey 
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To  get  a  measure  of  the  "experience  level"  of  the  WD  crew  force,  a 
"quantifying"  scheme  is  being  be  developed  that  calculates  an 
"experience  score"  for  every  WD  in  AWACS.  The  quantifying  scheme  will 
depend  on  factors  that  you  indicate  provide  valuable  indications  of 
individual  experience.  Please  give  your  undivided  consideration  to  each 
answer  you  provide.  Your  contributions  will  be  invaluable  in  managing 
new  personnel  acquisitions. 

Assume  you  must  pick  the  WD’s  for  the  crew  going  on  a  critical 
mission.  Further,  assume  you  have  no  personal  knowledge  of  the  WDs  you 
have  available  to  form  the  crew.  All  you  have  is  information  from 
various  databases  that  contain  career  information  on  each  candidate. 


PPRT  Ha). 

The  following  questions  show  a  scale  relating  to  the  importance  of 
various  data  elements.  These  data  elements  are  available,  and  may  be 
helpful  to  you  in  forming  the  best  crew  possible  from  the  available 
personnel.  With  the  situation  describe  above,  select  the  scale  value 
that  best  describes  the  value  you  would  place  on  the  having  access  to 
the  indicated  data  to  help  you  assess  the  candidates  experience.  The 
scale  values  correspond  to  the  following  descriptions; 

Data  in  this  area  is  _  in  selecting  the  best  men  for  this  job. 

1  —  not  useful 

2  —  some'^hat  useful 

3  —  useful 

4  —  very  useful 

5  —  exceptionally  valuable 

Note:  Make  your  assessment  based  only  on  the  data  element  being 
considered.  Do  not  make  your  assessment  to  the  other  data  elements 
discussed  in  the  survey. 

DATA  ELEJ'tNT 

1)  StanEval  ratings  on  last  three  check  rides 

Not  useful  1 -  2  3  -  4  -  5  very  valuable 

2)  Total  E-3  flying  hours 

Not  useful  1 -  2  3  -  4  -  5  very  valuable 

3)  Years  in  position 

Not  useful  1 -  2  3  -  4  -  5  very  valuable 


The  scale  values  descriptions  are  repeated  here  for  your  convenience. 
Data  in  this  area  is  _  in  selecting  the  best  men  for  this  Job. 

1  —  not  useful 

2  —  somev^iat  useful 

3  —  useful 

4  —  very  useful 

5  —  exceptionally  valuable 

Note:  Make  your  assessment  based  only  on  the  data  element  being 
considered.  Do  not  make  your  assessment  to  the  other  data  elements 
discussed  in  the  survey. 


4)  Instructor  qualified  at  some  time 

Not  useful  1 -  2  3  4  5  very  valuable 

5)  Number  of  E-3  missions  flown 

Not  useful  1 -  2  3  4  5  very  valuable 

6)  Number  of  exercise  missions  floun 

Not  useful  1 -  2  3  4  5  very  valuable 

7)  Ivkomber  of  Simulator  sessions 

Not  useful  1 -  2  3  4 - 5  very  valuable 

8)  Training  evaluations  for  courses  in  the  WD  training  sequence 

Not  useful  1 -  2  3  4  5  very  valuable 

9)  Number  of  controls  if  from  a  ground  controller  background 

Not  useful  1 -  2  3  4  5  very  valuable 


167 


Kb). 


In  this  portion  of  the  survey,  you  are  asked  to  conpare  the  data 
elements  in  the  previous  section  two  at  a  time.  Based  on  the  pair-wise 
comparison  stated  in  each  of  the  following  questions,  please  select  the 
data  element  you  most  value  in  selecting  the  WDs  for  the  crew  you  are 
forming.  Indicate  your  choice  by  ci'^cling  the  appropriate  choice  in  the 
center  column.  Try  to  answer  all  questions.  Ties  are  not  allowed. 

If  I  could  only  have  one  piece  of  information,  I  would  prefer  to  know 
*  each  candidate  UID's 


CIRCLE  I  or  II 


1) 

Years  in  position 

I 

or 

II 

Instructor  qual i f icat ion 
status  (all  of  career) 

2) 

Total  E-3  flying  hours 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

3) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Instructor  qualification 
status  (all  of  career) 

4) 

Years  in  position 

I 

or 

II 

Number  of  controls  if 
previously  a  ground 
controller 

5) 

Number  of  E-3  missions 
flo^.n 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

6) 

StanEval  rat ings  on 
last  three  check  rides 

I 

or 

II 

Training  evaluations 
for  all  courses  in  the 

WD  training  sequence 

7) 

Instructor  qualification 
status  (all  of  career) 

I 

or 

II 

Number  of  simulator 
sessions 

8) 

Total  E-3  flying  hours 

I 

or 

II 

Years  in  position 

9) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Total  E-3  flying  hours 

10) 

Total  E-3  flying  hours 

I 

or 

II 

Number  of  exercise 
missions  flown 

11) 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

I 

or 

II 

Number  of  controls 
if  previously  a  ground 
controller 
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If  I  ccxild  only  have  one  piece  of  information,  I  would  prefer  to  know 
each  candidate  WD's 


CIRCLE  I  or  II 


12) 

Instructor  qualification 
status  (all  of  career) 

I 

or 

II 

Number  of  E-3  missions 
flown 

13) 

Number  of  exercise 
missions  floun 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

14) 

Total  E-3  flying  hours 

I 

or 

II 

Number  of  E-3  missions 
flown 

15) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Number  of  exercise 
missions  flown 

16) 

Years  in  position 

I 

or 

II 

Number  of  simulator 
sessions 

17) 

Number  of  simulator 
sessions 

I 

or 

II 

Number  of  controls 
if  previously  a  ground 
control ler 

18) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Years  in  position 

19) 

Instructor  qualification 
status  (all  of  career) 

I 

or 

II 

Number  of  exercise 
sessions  flown 

20) 

Number  of  simulator 
sessions 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

21) 

Years  in  position 

I 

or 

II 

Number  of  E-3  missions 
flown 

22) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Number  of  controls  if 
previously  a  ground 
cont'^ol  ler 

23) 

Total  E-3  flying  hours 

I 

or 

II 

Instructor  qualification 
status  (all  of  career) 

24) 

Number  of  exercise 
missions  flown 

I 

or 

II 

Number  of  controls 
if  previously  a  ground 
control ler 
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If  I  could  only  have  one  piece  of  information,  I  would  prefer  to  know 
each  candidate  WD’s 


CIRCLE  I  or  II 


25) 

Number  of  E-3  missions 
flown 

I 

or 

II 

Number  of  simulator 
sessions 

26) 

StanEval  ratings  on 
last  three  check  rides 

I 

or 

II 

Number  o'  E-3  missions 
flown 

27) 

Years  in  position 

I 

or 

II 

Number  of  exercise 
missions 

28) 

Total  E-3  flying  hours 

1 

or 

II 

Number  of  simulator 
sessions 

29) 

StanEval  ratings  on 
last  three  check  rides 

1 

or 

II 

Number  of  simulator 
sessions 

30) 

Total  E-3  flying  hours 

I 

or 

II 

Number  of  controls  if 
previously  a  ground 
controller 

31) 

Instructor  qual i f icat ion 
status  (all  of  career) 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
training  sequence 

32) 

Number  of  E-3  missions 
flown 

I 

or 

II 

Number  of  exercise 
missions  flown 

33) 

Years  in  position 

I 

or 

II 

Training  evaluations  for 
all  courses  in  the  WD 
sequence 

34) 

Number  of  exercise 
missions  flown 

I 

or 

II 

Number  of  simulator 

sessions 

35) 

Number  of  E-3  missions 
f  1  own 

I 

or 

II 

Number  o'  controls  if 
previously  a  ground 
control ler 

36) 

Instructor  qualification 
status  (all  of  career) 

T 

or 

II 

Number  of  controls  if 
previously  a  ground 
control ler 
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FWT  II 


Please  take  a  mornents  to  indicate  factors  tha^  •-■o-'l':'  be 
valuable  indicators  of  a  WDs  experience.  If  you  choose  to  list  some 
factors,  please  limit  your  suggestions  to  those  that  you  think  are  at 
least  as  good  an  indicator  as  the  one  you  feel  is  the  best  in  the  set 
you  have  been  answering  questions  about. 

1.  _ 

2.  _ 

3.  _ 

4.  _ 

5.  _ 
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Tatj  B  —  Example  Transitivity  Examination 
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QUFSTION 

NUMBER 


PAIR 

COMPf^ED 


RESPCMDENT 

1 


PREFEREMCE  BY 
GROUP 


9 

A 

B 

A 

18 

A 

C 

A 

3 

A 

D 

A 

26 

A 

E 

A 

A  preferred 

15 

A 

F 

A 

all 

29 

A 

G 

A 

6 

A 

H 

A 

22 

A 

I 

A 

a 

B 

C 

C 

23 

B 

D 

D 

14 

B 

E 

E 

10 

B 

F 

F 

C,D,E,F,H,  I 

28 

B 

G 

B 

2 

B 

H 

H 

30 

B 

I 

I 

1 

C 

D 

D 

21 

C 

E 

C 

27 

C 

F 

C 

D,  I  >  C  >  E, 

16 

C 

G 

C 

33 

C 

H 

C 

4 

C 

I 

I 

12 

D 

E 

D 

19 

D 

F 

D 

7 

D 

G 

D 

I  >  D  >  E,F 

31 

D 

H 

D 

36 

D 

I 

I 

32 

E 

F 

F 

25 

E 

G 

E 

F,  I  >  E  >  G 

5 

E 

H 

E 

35 

E 

I 

I 

34 

F 

G 

F 

13 

F 

H 

F 

I  >  F  >  G,H 

24 

F 

I 

I 

20 

G 

H 

H 

H,  I  >  G 

17 

G 

I 

I 

11 

H 

I 

I 

I  >  H 

The  overall  preference  order  expressed  by  respondent  one  was  fully 
transitive,  and  is:  A>I  >D>C>r>E>H>B>G 
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Tab  C  -  SmaurtEdge  Ckitput 
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SmartEdge  was  exercised  in  four  different  modes  to  see  how 
different  the  results  were  for  tlie  oroject  problem.  These  run  modes 
were: 

1)  Additive  Model  —  Probabilistic 

2)  Additive  Model  —  Deterministic 

3)  Multiplicative  Model  —  Probabilistic 

4)  Multiplicative  Model  —  Deterministic 

All  four  applications  of  returned  the  same  ranking.  Further,  the 
differences  in  the  utility  values  differed  by  at  most  27.. 

In  this  case,  there  is  no  significant  difference  between  the 
additive  and  multiplicative  model  results.  Further,  the  results  were 
insensitive  to  the  uncertainty  in  attribute  level  set  for  the  initial 
"representative  WDs". 

The  edited  summary  reports  for  these  runs  follow.  The  editing 
ccnsists  of  renoving  Section  III  uhich  reports  on  differences  among 
multiple  decision  makers  (only  one  used)  and  the  inclusion  of  only  one 
set  of  appendices  (Section  IV)  for  the  four  runs,  since  the  information 
for  all  runs  remained  the  same. 
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SUM^Y  REPORT  CONTETvTTS 
FOR  ALL  CASES 


This  report  summarizes  the  results  of  PROBABILISTIC/ADDITIV/E!  as  folla^^: 

I  Introduction  and  Case  Specifications 

A.  The  Alternatives 

B.  The  Attributes 

C.  The  Decision  Makers 

D.  The  Analysis  Mode 

E.  The  Run  Option 

F.  The  Decision  Model 

II  Individual  Decision  Maker  Rankings 

III  Decision  Maker  Rankings  As  a  Group  (NOT  INCLUDED  —  ONLY  1  DECISION 
M=«ER) 

IV  appendices  (INCLUDED  ^TER  THE  ANALYSIS  OF  PLL  FOUR  RUNS) 

A.  The  Attribute  State/Distribution  Inputs 

B.  The  Preference  Inputs  for  Each  Decision  Maker 
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SECTION  I 

INTRODUCTION  CASE  SPEC  IE  I  CAT  IONS 


This  Section  presents  the  spec i f icat ions  for  the  decision  making  case 
def ined. 


A.  The  Alternatives  are: 


Alter at ive 

1  = 

1 

Alter at ive 

2  = 

2 

A1 ter at ive 

3  = 

3 

Alter at ive 

4  = 

4 

Alter at ive 

5  = 

5 

A1 ter at ive 

fe  = 

6 

Alter at ive 

7  = 

7 

Alter at ive 

8  = 

8 

Alterative 

9  = 

9 

Alter at ive 

10  = 

10 

Alter at ive 

11  = 

BEElviAROLJND 

Alterative 

12  = 

SEENSOMEl 

Alter at ive 

13  = 

SEENSGME2 

Alterative 

14 

OLDDOG 

B.  The  Attributes  are: 

Attribute  1  =  StanEval  Ratings 
Attribute  2  ~  ttExerc iseMissions 


C.  The  Decision  Makers  are: 

Individual  1  =  DM2 

D.  Mode  of  Analysis  Selected  is:  Detailed  Analysis  Mode 

E.  Run  Option  for  Uncertainty  is:  Probabilistic 

E.  Type  of  Decision  Model  is:  Sinple  Linear  (Additive)  Model 
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SECTION  II 

INDIVIDUAL  DECISION  (*V:^KER  RA^KINGS 


This  Section  presents  the  results  of  a  multi-attribute  decision  analysis 
for  the  Case  PROBABILISTIC/ADDITIVE  with  rankings  for  each  decision 
maker. 

Rankings  for  Decision  Maker:  DM2 

Alternative  Value  Rank 

1  0.0776  14 

+/-  0.0327 

2  0.1261  13 

+/-  0.0457 

3  0. 1633  12 

+/-  0.0596 

4  0.2521  9 

+/-  0.0736 

5  0.3263  a 

+/-  0.0767 

6  0.3388  7 

+/-  0.0988 

7  0.4429  6 

-*■/-  0.1227 

8  0. 4607  5 

+/-  0.1254 

9  0.6032  3 

+/-  0. 1399 

10  0.6768  2 

+/-  0.1193 

BEET'i^mJND  0.5342  4 

+/-  0.0005 

SEENSOMEl  0.2465  10 

+/-  0.0002 

SEENStr^  0.1758  ll 

+/-  0.0001 

□LDDCX3  0.9088  1 

+/-  0.0004 

The  rankings  are  in  Agreement  at  a  95  percent  significance  level. 
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SUMil^Y  REPORT 

FOR  CASE:  DETB^MINISTIC/ADDITIVE 
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SECTION  I 

INTRODUCTION  PND  CPGE  SPECIFICATIONS 


This  Section  presents  the  specifications  for  the  decision  making  case 
def ined. 

A.  The  Alternatives  are: 


Alterat ive 

1 

1 

Alter at ive 

2  = 

2 

Alterat ive 

3  = 

3 

Alterat ive 

4  = 

4 

Alterat ive 

5  ^ 

5 

Alterat ive 

S  = 

6 

Alterat ive 

7  = 

7 

Alterat ive 

8  = 

8 

Alterat ive 

9  = 

9 

A1  ter at ive 

10  = 

10 

A1 ter at ive 

11  = 

BEEm^OUND 

Alterat ive 

12  = 

SEENSOMEl 

A1 ter at ive 

13  = 

SEENS0ME2 

A1 ter at ive 

14  = 

□LDDQG 

B.  The  Attributes  are: 

Attribute  1  =  Stantval  Ratings 
Attribute  2  =  #Exerc iseMissions 

C.  The  Decision  Makers  are: 

Individual  1  =  DM2 

D.  Mode  of  Analysis  Selected  is:  Detailed  Analysis  Mode 


E.  Run  Option  for  Uncertainty  is;  Deterministic 


F.  Type  of  Decision  Model  is:  Simple  Linear  (Additive)  Model 
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SECTION  II 

INDIVIDUAL  DECISION  r-V^KER  RATvKINGS 


This  Section  presents  the  results  of  a  multi-attribute  decision  analysis 
for  the  Case  DETERMINISTIC/ADDITI'»/t  with  rankings  for  each  decision 
maker. 

Rankings  for  Decision  Maker:  DM2 


Alternat ive 

Value 

Rank 

1 

0.0793 

14 

2 

0. 1221 

13 

3 

0. 1650 

12 

4 

0.2521 

9 

5 

0.3259 

a 

6 

0.3532 

7 

7 

0.4262 

6 

8 

0.4550 

5 

9 

0.6055 

3 

10 

0.6616 

2 

BEEI^mDUND 

0.5342 

4 

SEENSQMEl 

0.2465 

10 

SEENSOTES 

0. 1758 

11 

OLDDOG 

0.9088 

1 

The  rankings  are  in  Agreement  at  a  95  percent  significance  level. 
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SUMTWY  REPCFT 

FOR  CASE:  PROBABILISTIC/MJLTIPLICATIVE 
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SECTION  I 

INTRODUCTION  AND  CASE  SPECIFICATIONS 


This  Ejection  presents  the  specifications  for  the  decision  making  case 
defined. 

A.  The  Alternatives  are: 


Alter at ive 

1  = 

1 

A1 ter at ive 

2  = 

2 

Alter at ive 

3  = 

3 

Alterative 

4  = 

4 

Alter at ive 

5  = 

5 

Alterative 

6  = 

S 

A1 ter at ive 

7  = 

7 

Alterative 

8  = 

8 

A1 ter at ive 

9  = 

9 

Alterative 

10  = 

10 

Alter at ive 

11  = 

BEEN(=tf?0UND 

A1 ter at ive 

12 

SEENSOrt  1 

Alter at ive 

13  = 

SEENS0rt2 

Alter at ive 

14  = 

OLDDOG 

B.  The  Attributes  are: 

Attribute  1  =  StanEval  Ratings 
Attribute  2  =  #Exerc iseMissions 

C.  The  Decision  Makers  are: 

Individual  1  =  DM2 

D.  Mode  of  Analysis  Selected  is:  Detailed  Analysis  Mode 


E.  Run  Option  for  Uncertainty  is:  Probabilistic 


F.  Type  of  Decision  Model  is:  Detailed  Non-Linear  (Multiplicative) 
Model 
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SECTION  II 

INDIVIDUAL  DECISION  MAKER  RANKINGS 

This  Section  presents  the  results  of  a  multi-attribute  decision  analysis 
for  the  Case  PROBABILISTIC/MULTIPLICATIVE  with  rankings  for  each 
decision  maker. 

Rankings  for  Decision  Maker:  DM2 

Alternative  Value  Rank 

1  0.0739  14 

+/-  0.0312 

2  0.1205  13 

+/-  0.0437 

3  0. 1565  12 

+/-  0.0573 

4  0.2420  9 

+/-  0  0700 

5  0.3146  a 

+/-  0.0741 

6  0.3285  7 

+/-  0.0964 

7  0.4308  6 

+/-  0.1210 

a  0. 4496  5 

+/-  0.1235 

9  0.5941  3 

+/-  0.1399 
10  0.6657  2 

+/-  0.1191 

BEErvmDUND  0.5263  4 

+/-  0.0004 

SEENSOMEl  0.2404  10 

+/-  0-0001 

SEENSaME2  0. 1689  1 1 

+/-  0.0002 

□LDDOG  0.8800  1 

+/-  0.0008 

The  rankings  are  in  Agreement  at  a  95  percent  significance  level. 
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SUrtlf^RY  REPORT 

FOR  CASE:  DETERMINISTIC/MULTIPLICATIVE 
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SECTION  I 

INTRODUCTION  AND  CASE  SPECIFICATIONS 


This  Section  presents  the  specifications  for  the  decision  making  case 
defined. 


A.  The  Alternatives  are; 


A1 ter at 
Alter at 
Alter at 
Alter at 
Alter at 
Alter at 
A1 ter at 
Alter at 
A1 ter at 
Alter at 
Alter at 
Alter at 
Alter at 
Alter at 


ive  1  = 
ive  2  = 
ive  3  = 
ive  4  = 
ive  5  = 
ive  6  = 
ive  7  = 
ive  B  = 
ive  9  = 
ive  10 
ive  11 
ive  12 
ive  13 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

BEENAROU'JD 

SEENSOMEl 

SEENS0ME2 

OLDDOG 


B.  The  Attributes  are; 

Attribute  1  =  StatiEval  Ratings 
Attribute  2  =  ttExerc iseMissions 


C.  The  Decision  Makers  are; 
Individual  1  =  DM2 


D.  Mode  of  Analysis  Selected  is;  Detailed  Analysis  Mode 


E.  Run  Pption  for  Uncertainty  is;  Deterministic 

F.  Type  of  Decision  Model  is:  Detailed  Non-Linear  (Multiplicative) 
Model . 
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SECTION  II 

INDIVIDUAL  DECISION  MAKER  RANKINGS 


This  Section  presents  the  results  of  a  multi-attribute  decision  analysis 
for  the  Case  DETERFilNISTIC/MULTIPLICATIVE  with  rankings  for  each 
decision  maker. 

Rankings  for  Decision  Maker:  DM2 


A1 ternat i ve 

Value 

Rank 

1 

0.0756 

14 

0.1167 

13 

3 

0. 1580 

12 

4 

0.2421 

9 

5 

0.3143 

a 

6 

0.3426 

7 

7 

0.4139 

6 

8 

0.4435 

5 

9 

0.5963 

3 

10 

0.6503 

2 

BEElv*=ROUND 

0.5263 

4 

SEENSOMEl 

0.2404 

10 

SEENSaME2 

0. 1689 

11 

0LDDCX3 

0.8800 

1 

The  rankings  are  in  Agreement  at  a  S6  percent  significance  level. 
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SECTION  IV 
DATA  INPUTS 


This  Ejection  presents  the  inputs  for  Case  PROBABILISTIC/ADDITIVE  used  to 
corpute  the  rankings. 

A.  The  Attribute  State/Distribut ion  Inputs: 
attribute  States  for  Alte'^native  :  1 


Cumulative  Distribution 

tor  StanEval 

State  CDT 

State  CDT 

State 

CDF 

1.0000  0.00 

2.0000  0.50 

3.0000 

1.00 

2.0000  0.2b 

2.0000  0.75 

Cumulative  Distribution 

for  #Exerc iseMissions 

State  CDF 

State  CDF 

State 

CDF 

0.0000  0.00 

2.0000  0.50 

3.0000 

1.00 

0.0000  0.2b 

3.0000  0.75 

Attribute  States  for  Alternative  :  2 


Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

1.0000  0.00 

3.0000  0.50 

3.0000 

1.00 

2.0000  0.2b 

3.0000  0.7b 

Cumulative  Distribution 

for 

#Exerc iseNissions 

State  CDF 

State  CDF 

State 

CDF 

0.0000  0.00 

4.0000  0.50 

5.0000 

1.00 

2.0000  0.2b 

4.0000  0.75 

Attribute  States  for  Alternative  :  3 


Cumulative  Distr ibut icxi 

for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

2.0000  0.00 

3.0000  0.50 

4,0000 

1.00 

2.0000  0.25 

4.0000  0.75 

Cumulative  Distribution 

for 

♦♦Exerc  iseflissions 

State  CDF 

State  CDF 

State 

CDF 

2.0000  0.00 

5,0000  0.50 

7.0000 

1.00 

4.0000  0.2b 

5.0000  0.75 
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Attribute  States  for  Alternative  ;  4 


Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF 

St  ate 

CDF 

3.0000  0.00 

4.0000  0,50 

6.0000 

1.00 

3.0000  0.25 

5.0000  0,75 

Cumulative  Distribution 

for 

ttExerc i seM i ss i ons 

State  CUf 

State  CDF 

State 

CDF 

4.0000  0.00 

6.0000  0,5vy 

8.0000 

1.00 

5.0000  0.25 

7.0000  0.75 

Attribute  States  for  Alternative  :  5 


Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

3.0000  0.00 

5.0000  0.50 

7,0000 

1,00 

4.0000  0.25 

6.0000  0.75 

Cumulative  Distribution 

for 

#Exerc iseMissions 

State  CDF 

State  CDF 

State 

CDF 

7.0000  0.00 

8.0000  0.50 

10.0000 

1.00 

7.0000  0.25 

10.0000  0.75 

Attribute  States  for  Alternative  :  6 


Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

3.0000  0.00 

5.0000  0.50 

8.0000 

1.00 

4.0000  0.25 

6.0000  0.75 

Cumulative  Distribution 

for 

ttExerc iseMissions 

State  CDF 

State  CDF 

State 

CDF 

7.0000  0,00 

14.0000  0.50 

18.0000 

1.00 

9,0000  0.25 

14.0000  0.75 

Attribute  States  for  Alternative  :  7 


Cumulative  Distribution  for  StanEval 

State  CDF  State  CDF  State  CDF 
3.0000  0.00  6.0000  0.50  8.0000  1.00 
5.0000  0.25  8.0000  0.75 


Cumulative  Distribution  for  ttExerc iseMissions 

State  CDF  State  CDF  State  CDF 
7.0000  0.00  14.0000  0.50  18.0000  1.00 
7.0000  0.25  14.0000  0.75 
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Attribute  States  for  Alternative  :  8 


Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

4.0000  0.00 

6.0000  0.50 

9.0000 

1.00 

5.0000  0.25 

8.0000  0.75 

Cumulative  Distribution 

for 

#Exerc iseMissions 

State  CDF 

State  CDF 

State 

CDF 

10.0000  0.00 

14.0000  0.50 

IS. 0000 

1.00 

14.0000  0.25 

18.0000  0.75 

Attribute  States  for  Alternative  :  9 

Cumulative  Distribution 

for 

StanEval 

State  CDF 

State  CDF' 

St  ate 

CDF 

4.0000  0.00 

8.0000  0.50 

11.0000 

1.00 

7.0000  0.25 

9.0000  0.75 

Cumulative  Distribution 

for 

#Exerc i seM i ss i ons 

State  CDF 

State  CDF 

State 

CDF 

14.0000  0.00 

18.0000  0.50 

21.0000 

1.00 

14.0000  0.25 

21.0000  0.75 

Attribute  States  for  Alternative  ;  10 

Cumulative  Distribution  for 

StanEval 

State  CDF 

State  CDF 

State 

CDF 

7.0000  0.00 

8.0000  0.50 

12.0000 

1.00 

8.0000  0.25 

9.0000  0.75 

Cumulative  Distribution  for 

fJExer  c  i  seM  i  ss  i  ons 

State  CDF 

State  CDF 

State 

CDF 

14.0000  0.00 

18.0000  0.50 

21.0000 

1.00 

14.0000  0.25 

18.0000  0.75 

Attribute  States  for  Alternative  :  BEENAROUND 

Point  Value  for  StanEval 

=  7.0000 

Point  Value  for  #Exerc iseMissions  = 

18.0000 

Attribute  States  for  Alternative  ;  SEENSOMEl 

Point  Value  for  StanEval  =  3.0000 

Point  Value  for  fCxerc iseMissions  =  18.0000 
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Attribute  States  for  Alternative  :  SEENS0ME2 


Point  Value  for  StanEval  -  3.00CXD 

Point  Value  for  #Exerc iseMissions  =  7,0000 


Attribute  States  fo»  Alternative  ;  OLDEXDG 


Point  Value  for  StanEval  =  12.0000 

Point  Value  for  flExerc  iseMissions  =  10.0000 


B.  The  Preference  Inputs  for  Each  Decision  Maker 
Attribute  Preferences  for  Individual:  DM2 


Scaling  Constraints  (Heights)  For  Each  Attribute  Are: 

StanEval:  0.8632 

MExerc iseMissions:  0.1368 

The  Utility  Function  for  Each  Attribute  Are: 
ATTRIBUTE;  StanEval 


State  Utility 

State 

Util  ity 

State 

Utility 

1.0000  0.00 

7.0000 

0.50 

12.0000 

1.00 

ATTRIBUTE:  #Exerc iseMissions 

State  Utility 

State 

Util  ity 

State 

Utility 

0.0000  0,00 

15.0000 

0.50 

21.0000 

1.00 
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